## Summary
Since we are enabling `useModernStrictMode` flag internally, to make
sure the internal testing of half StrictMode doesn't suddenly break,
this PR makes sure it also works with `useModernStrictMode` true.
## Test plan:
Manually set `useModernStrictMode` to true.
`yarn test ReactOffscreenStrictMode-test -r=www-modern --env=development
--variant=true`
`yarn test ReactStrictMode-test.internal -r=www-modern --env=development
--variant=true`
## Summary
This was not exposed as a dynamic flag in the build for facebook www. By
adding it, we'll be able to roll this out incrementally before cleaning
up this code altogether.
## How did you test this change?
`yarn build`
Before changes, `disableSchedulerTimeoutInWorkLoop` flag is not included
in ReactDOM-* build output for facebook www. Afterwards, it is included.
Ports `MergeConsecutiveBlocks` to Rust. This was a tricky one: as we iterate
through the blocks _if_ the block ends up being merged with its predecessor we
need to consume it and modify its predecessor block (ie, mutating two things
from the same data structure - shared mutation!). But if we _don't_ need to
merge, then we need to not drop the current block. Ie, we sort of need to
conditionally take ownership of the current block during iteration and put it
back.
I added a `BlockRewriter` helper type for this which has a helper to iterate
safely. It calls the iterator lambda, moving blocks one at a time into the
lambda. The lambda returns either `Keep(block)` to give the block back and keep
it or `Remove` to tell the rewriter to drop the block. Thanks to making the
Blocks data structure hold `Option<Box<BasicBlock>>` items, "moving" the block
actually just means nulling out the option and conceptually giving ownership of
the pointer to the callback - the data itself never moves.
This involved creating a custom `Blocks` wrapper type, which i had been putting
off doing. This cleans up a bunch of other logic around traversing blocks.
This PR adapts the `Diagnostic` type and helpers from Relay Compiler to Forget.
The main changes are:
* Removing some fields it doesn't seem we'll use for a while, if ever (like
machine-readable arbitrary key/value data)
* Switching from Relay Compiler's `Location` type to our SourceRange type
* Using the severity enum previously established in the `forget_build_hir`
crate, with Todo/Unsupported/InvalidSyntax/InvalidReact/Invariant variants
* Adding support for translating our `Diagnostic` into a `miette::Diagnostic` so
we can use miette's pretty printing
With the new Diagnostic type in place i updated the existing build_hir code to
use it and confirmed that the errors are now even nicer (when we attach extra
data to annotate labels):
<img width="860" alt="Screenshot 2023-07-14 at 3 10 00 PM"
src="https://github.com/facebook/react-forget/assets/6425824/9d29425a-938b-4872-b999-aa174a3c329a">
This addresses (or brings us closer to addressing) many of your comments on the
last diagnostics PR, @poteto!
Per the title, this updates the main readme file with a guide to contributing to
the Rust compiler, and ensures that we have a brief description of every crate
in local readme.md files. The `forget_hir` one is the most extensive and
describes the high-level design of the HIR.
I've primarily used what Cargo calls virtual workspaces, where the top-level
Cargo.toml just lists a bunch of packages and they each have their own
dependencies. This is fine, but i've noticed that more repos are using real
workspaces and they offer a bunch of benefits. You define the dependencies in
the top-level Cargo.toml and then can easily refer to them from multiple crates,
ensuring all the versions match up. It makes it easier to refer to other crates
in the workspace, too, because you define the path once at the root, then every
other crate can just say `forget_foo = { workspace = true }`.
I also renamed all the crates to be prefixed with `forget_`, in some cases
removing the redundant `hir` name, So `hir-optimization` became
`forget_optimization`, `hir-ssa` became `forget_ssa`. Also note the switch from
hyphenated names to underscores everywhere, since at the end of the day you have
to write the name with underscores in source code.
I also deleted the demo crate that i started with since we don't need it
anymore.
And finally, i added an explicit publish = false to all the crates just to
prevent mistakes.
This is a quick "good enough" first pass at computing function expression
context variables. It definitely needs to be overhauled, but it's enough to make
a lot of common cases work correctly.
First, this PR adds a hand-rolled Visitor trait for `estree`. Long-term that
should probably be code-generated, but there are some subtleties to it such as
the `visit_lvalue(callback)` helper which has to be wrapped around various calls
(or we need some other way to distinguish identifiers within lvalues from
identifiers within rvaluess). So for now it makes sense to hand-roll it until we
are more confident in exactly how it should work.
Given that visitor, i was able to port part of the existing
`gatherCapturedDeps()` to Rust with some modifications. Note that we assume
_something_ has run name resolution on the estree to match up identifiers to the
declaration they refer to. But we don't store information about parent scopes so
we can't walk up to check where things were defined. Instead we do the
following:
* Build up a list of all referenced identifiers
* Also build up a set of bindings defined in the function itself
* After visiting, filter our the first list to only include identifiers not
defined by the function itself, and to eliminate duplicates.
This covers the majority of cases: the most obvious gap is nested function
expressions though actually that should just work.
Updates eliminate_redundant_phis and constant_propagation to recurse into
function expressions. I also realized there was a bug in EliminateRedundantPhis
in which we wouldn't traverse into function expressions encountered after
finding a back edge, so i fixed that logic in both versions.
Updates `enter_ssa()` to recurse into function expressions. In the TS compiler
we use a single Builder instance and copy (references to) the function
expression's blocks into the builder. That type of sharing just does not play
nicely with Rust. But... we don't need to do that! We already know the context
variables of the function expression, so we can lookup each of them to find
their re-mapped identifier, and set that as the starting state for the entry
block of the function expression. That lets us use normal recursion and
otherwise not share any information between the outer and inner builders.
Of course to make this work we actually have to populate Function.context, but
the algorithm _should_ work.
Start of function expression support:
* Basic structure for representing function expressions in the HIR
* Printer support
* swc -> estree -> hir conversion for function expression _bodies_. Dependencies
and context are not handled yet.
Makes the same improvement to constant propagation in both the JS and Rust
versions. The core algorithm only populates phi variables if all operands have a
known value (no back edges) and all those values are the same: this allows us to
propagate constants in most cases and simply punts on handling propagating
values that are affected by loops. However, since we collapse if statements into
gotos when the test condition is a constant, there can be cases where a phi that
originally existed will be pruned out:
```
// bb0
let x1;
if (true) {
// bb1
x2 = 1;
} else {
// bb2
x3 = 2; // this block becomes unreachable
}
// bb3
x4 = phi(bb1: x2, bb2: x3); // this phi will get pruned s.t. x4 = x2 = 1
return x4;
```
However, the algorithm doesn't prune phis until _after_ applying constants, so
currently we would see this phi node w different inputs and not propagate a
constant for the final usage of x, even though it will clearly be `1`.
The change is to make constant propagation use fixpoint iteration, iterating so
long as terminals changed on the previous iteration. If no terminals change the
algorithm completes in a single pass, but if terminals do change then we update
phis and continue. As you can see from the new test case this allows us to find
arbitrary length sequences of values and terminals that can be pruned.
Ports constant propagation to Rust. The algorithm is broadly similar to the TS
version, and most of the differences come from the slightly different HIR data
model (operands are instruction indices not identifier ids). What this means is
that the Constants map that we build up is really only used for variables that
existed in the original program, and only comes into play with instructions like
LoadLocal and StoreLocal. Other instructions such as Binary just look up their
operands directly, ie they load the referenced instruction to check if both
left/right are primitives.
Note that with SSA form and the index-based operands we could actually get rid
of StoreLocal/LoadLocal completely, which would further simplify constant
propagation. However:
* we'd need to add a Phi instruction kind, not a big deal but it diverges even
more
* more importantly, it makes it super hard to implement LeaveSSA
That second point is a deal-breaker so unless someone has a great idea for how
to exit SSA form without having Load/Stores, let's keep them.
This is a nearly 1:1 port of EliminateRedundantPhis to Rust, the algorithm is
identical and all differences are superficial. There are few things missing (an
invariant instead of a panic in one place, recursing into function expressions)
but the Rust version is still going to end up shorter despite keeping all the
comments.
This is a first pass at porting EnterSSA to Rust. First pass in the sense that
it's hard to fully test it, and also in the sense that we'll likely figure out
even better ways to work w the HIR as we iterate. Oh and i didn't do recusing
into function expressions yet, since we can't even represent function
expressions yet, and that may require modifying the design a bit (though i have
an idea that i think will work, which is for the Builder to have an optional
parent. When we encounter a block with no predecessors, we check the parent. I
_think_ this will make all the borrowing "just work").
A few notes:
Rust's compilation model parallelizes and incrementally computes at the crate
granularity, so builds are faster if we split up our code into more but smaller
crates. Setting up a clean dependency graph can dramatically improve build
performance too. For example, to run the `fixtures` tests we can build `estree`
and `hir` in parallel, then once those build _all_ of our other crates can be
built in parallel until we get to `fixtures` which depends on the everything
else. The various passes don't have any build dependencies on each other so they
can build in parallel. Hence the new code for SSA stuff is in a separate
`hir-ssa` crate. We should similarly group other passes (approximately one crate
per folder in the babel-plugin-react-forget/src/ directory, eg SSA, Inference,
Optimization, etc).
Second, shared mutable ownership can be modeled in Rust but requires wrappers
such as `Rc<RefCell<>>`. It's generally more efficient and more idiomatic to
rethink the data model and algorithm. For EnterSSA, the Builder object holds a
reference into the HIR that it only ever reads, and the pass (which drives the
builder) also holds a reference into the HIR, which it mutates. The previous PR
split up Blocks and Instructions, and the value of that is more apparent in
`enter_ssa()`. The Rust equivalent of the builder holds a _shared_ (immutable)
reference to just the HIR's blocks, while the pass (driving the builder) holds a
_unique_ (mutable) reference to just the HIR's instructions. This lets us keep
the overall feel of the algorithm while keeping Rust happy.
Also note that the other change — to making operands be InstrIx indices into the
instructions array — means that the SSA logic is simpler. Most instructions
don't have to be visited at all, since they don't deal with loads/stores.
Terminals also don't need to be visited, since they reference instructions, not
identifiers. The Phi concept seems to just work too.
I also updated the printer to print predecessors and phis.
## Summary
`scheduler.yield` is entering [Origin Trial soon in Chrome
115](https://chromestatus.com/feature/6266249336586240). This diff adds
it to `SchedulerPostTask` when scheduling continuations to allow Origin
Trial participation for early feedback on the new API.
It seems the difference here versus the current use of `postTask` will
be minor – the intent behind `scheduler.yield` seems to mostly be better
ergonomics for scheduling continuations, but it may be interesting to
see if the follow aspect of it results in any tangible difference in
scheduling (from
[here](https://github.com/WICG/scheduling-apis/blob/main/explainers/yield-and-continuation.md#introduction)):
> To mitigate yielding performance penalty concerns, UAs prioritize
scheduler.yield() continuations over tasks of the same priority or
similar task sources.
## How did you test this change?
```
yarn test SchedulerPostTask
```
This change is motivated by starting to explore porting EnterSSA to Rust. It's a
good medium complexity pass and quickly demonstrates why a direct port of our
existing data model and algorithms won't work so well. For examples just these
first lines at the top of the transform create multiple references to the
function body/blocks:
https://github.com/facebook/react-forget/blob/58da89888eabde17ede649d348b307ee01fc15e1/forget/packages/babel-plugin-react-forget/src/SSA/EnterSSA.ts#L230-L231
But more generally it's always felt wrong that instructions have an LValue that
isn't really used. So here i'm exploring making operands a newtype index into a
single instructions array (shared for the entire function), and using different
types for identifier references in SetLocal and LoadLocal. Incidentally this
also declutters the printed HIR quite a bit.
I'm not going to land this until i actually finish enter_ssa() and some other
passes. We definitely need to balance fidelity to the existing code (to
facilitate porting) with using idiomatic Rust (to facilitate porting in the
sense of not fighting the borrow checker).
This mode improves compilation and makes optimizations easier, let's make it the
default. I previously confirmed that enabling this mode didn't affect output
when synced internally, and I'll do that again before landing the PR.
I added this as a quick workaround, since we didn't support unused
logical/conditional expression statements. Now that we handle them we don't need
InstructionValue::ExpressionStatement anymore. I found this when porting our
lowering to Rust.
The previous PRs to make `estree` use codegen broke the swc->estree and
estree->hir conversions. This PR updates those conversions so everything builds
now.
The overall goal of this workstream is to have a Rust representation of ESTree
that we can use as the input and output of the compiler. In Rust environments we
can convert between the native AST of SWC or OXC and ESTree, and when invoked
from JavaScript we can serialize to/from ESTree-compliant JSON. Given that our
first target is to plug into a JS-based compilation toolchain, we need to have a
working serialization to/from ESTree JSON. The point of the codegen-based
`estree` crate is to allow us to model estree as ergonomic, idiomatic Rust (to
make consuming it in code easier) while also allowing us to serialize to/from
spec-compliant ESTree. This PR flushes out one remaining piece.
Updates our estree codegen to generate a custom `Deserialize` implementation
instead of using the derived one from serde. ESTree has some enums whose
variants are themselves enums, for example we have
```rust
enum ModuleItem {
ImportExportDeclaration(ImportExportDeclaration),
Statement(Statement),
}
enum ImportExportDeclaration { ... }
enum Statement { ... }
```
This sort of works with serde's derive implementation: you have to use the "tag"
representation for the inner enums, and an "untagged" representation for the
outer one (ModuleItem). The problem is that with an untagged representation,
serde doesn't know what type of data it's expecting. All it can do is go one by
one and try to parse the data as the first variant (eg ImportExportDeclaration)
then the next one (Statement) and fail when it gets to the end of the list. If
the data isn't valid for any reason, deserialization will fail with a "not a
valid ModuleItem" error. That's true but not helpful, especially if you're
developing estree, are confident that the input json is valid, and need to
figure out where you messed up the definition. It's also not helpful as an
end-user if you're not sure your input json is valid.
So this PR updates our codegen to emit a custom derive implementation that is
identical for both regular enums (like Statement) and recursive ones (like
ModuleItem). We first extract the tag to know what type the value is, then
deserialize exactly as that type. So in the above case, rather than have to
first try parsing every ModuleItem as an ImportExportDeclaration and then fall
through to statement, we just decode the tag (`type` in our case, for example
say it's an "ForStatement"), then deserialize directly as that type (eg, as
ForStatement), then wrap it in the enum variant (ModuleItem::Statement(...)).
For recursive enums like ModuleItem we add an extra wrapper as necessary.
The end result is that we get much more precise errors and deserialization is
more efficient: we always decode just the tag, then as exactly that type.
Note that our serialization is also not perfect right now, because we don't
always emit the `type` key. Serde only emits it when a value appears in an enum.
We can similarly generate custom serializers for all our types to always emit
the tag. That will be straightforward when it's necessary. The current PR was
more of a blocker, because it was really hard to figure out mistakes in the
estree definition given the ambiguous errors. Thanks to this PR we now get
precise errors along the lines of "unknown type `JSXElement`" which are easy to
resolve.
Uses the `syn` crate, which can parse various Rust syntax forms, to parse the
`type` field from json schema description. This allows us to describe complex
types like `"type": "Vec<Option<ArrayElement>>"` directly, rather than requiring
flags like nullable, plural, and nullable_item. The main flag that i'm keeping
is "optional", which is used to indicate when the field itself (not the value)
is optional.
Adds some parts of the ES2015 spec, such as imports and ForOfStatement. This is
enough to get a few more fixtures compiling. The last one uses JSX which I
haven't defined yet.
This is meant to replace the initial `estree` crate with a version that is
generated from a JSON description of ESTree. The idea is to make it easy to
experiment with slightly different representations to balance ergonomic usage of
the data at runtime with serialization compatible with ESTree spec. The JSON
schema looks like this (somewhat abbreviated):
```
{
// Objects are struct types that don't have a `type` and can't appear as an enum
variant
objects: {
Position: {
line: {type: "NonZeroU32"},
column: {type: "u32"}
},
...
},
// Nodes are struct types with a `type` and which can appear as enum variants
(statements, expressions, patterns, etc)
nodes: {
ArrayExpression: {
elements: {
type: "Expression",
plural: true,
nullable_item: true,
}
}
...
},
// Categories of nodes with multiple variants, represented as enums.
// Can be recursive, eg ForInit can be VariableDeclaration or Expression,
// where Expression is also an enum
enums: {
Expression: [
"ArrayExpression",
...
]
},
// Simple enums which have a corresponding string value. Used primarily for
operators (binary/unary/logical/etc)
// but also for things like variable declaration kind (var/const/let)
operators: {
BinaryOperator: {
Plus: "+",
Instanceof: "instanceof",
}
}
}
```
The core estree files are now generated using Cargo's build script mechanism.
Right now i only defined the types and fields from ES5, so i'll have to flush
out the rest of the modern JS spec and extensions like JSX, TypeScript, and
Flow. But already the for-statement example works, showing that this approach
can handle complex cases such as unions of types or other unions (ForStatement
initializer is tricky bc it can be a VariableDeclaration or an Expression - that
works now!).
This is still WIP a bit - now that the ESTree definition is more precise i can
go back and clean up some other code (have to, because the swc -> estree
conversion needs some tweaks now).
Until now i've freely used `panic!`, `unwrap()`, and friends for "error
handling". This PR switches to consistently returning `Result` within the HIR
builder, using a structured error representation that exploits helpers from
`thiserror` and `miette` crates. Miette has a super graphical formatter for
diagnostics as you can see in the screenshot (also see the
[repo](https://docs.rs/miette/5.9.0/miette/index.html)).
This is just a first pass and we'll need to flush out the error handing story
more. Two obvious directions to go next:
* Make HIR construction error-tolerant, so that it can find as many errors as
possible at once rather than failing on the first error. We did this in Relay
Compiler as well, and we can likely borrow some of its helpers.
* Decouple from `miette`. It's very nice but less flexible than I'd like. We can
define our own more generic diagnostic type that contains structured data, then
have a generic conversion mechanism into a miette type so we can use their
display logic.
<img width="789" alt="Screenshot 2023-07-06 at 3 55 38 PM"
src="https://github.com/facebook/react-forget/assets/6425824/e1f1ed4b-5188-4af5-9af4-8f6c5c345023">
Implements support for `ForStatement` from swc -> estree -> hir, flushing out
more of the HIR representation and porting pieces from HIRBuilder as necessary.
We already did this for Server References on the Client so this brings
us parity with that. This gives us some more flexibility with changing
the runtime implementation without having to affect the loaders.
We can also do more in the runtime such as adding `.bind()` support to
Server References.
I also moved the CommonJS Proxy creation into the runtime helper from
the register so that it can be handled in one place.
This lets us remove the forks from Next.js since the loaders can be
simplified there to just use these helpers.
This PR doesn't change the protocol or shape of the objects. They're
still specific to each bundler but ideally we should probably move this
to shared helpers that can be used by multiple bundler implementations.
## Summary
as we began [discussing
yesterday](https://github.com/facebook/react/pull/27056#discussion_r1253282784),
`SuspenseList` is not actually stable yet, and should likely be exported
with the `unstable_` prefix.
the conversation yesterday began discussing this in the context of the
fb-specific packages, but changing it there without updating everywhere
else leads to test failures, so here the change is made across packages.
## How did you test this change?
```
yarn flow dom-browser
yarn test
```
When selecting a package variant from an export map we should favor node
over edge-light
edge-light represents a runtime with some minimal set of web apis
generally found across edge runtimes. However some environments might be
both edge-light compatible and node compatible and (node is adding many
web APIs) and when both conditions exist we want to favor the node
implementations. A followup to this change will add the web streams APIs
to Flight and Fizz so the node version exports the same interfaces for
web streams that edge does in addition to the node specific
implementations.
Andrew Clark
Tianyu Yao
Noah Lemen
Lauren Tan
Joe Savona
Ruslan Lesiutin
Steven
Sebastian Markbåge
Josh Story
Sathya Gunasekaran