[rust] Custom deserialization for more precise errors

The overall goal of this workstream is to have a Rust representation of ESTree 
that we can use as the input and output of the compiler. In Rust environments we 
can convert between the native AST of SWC or OXC and ESTree, and when invoked 
from JavaScript we can serialize to/from ESTree-compliant JSON. Given that our 
first target is to plug into a JS-based compilation toolchain, we need to have a 
working serialization to/from ESTree JSON. The point of the codegen-based 
`estree` crate is to allow us to model estree as ergonomic, idiomatic Rust (to 
make consuming it in code easier) while also allowing us to serialize to/from 
spec-compliant ESTree. This PR flushes out one remaining piece. 

Updates our estree codegen to generate a custom `Deserialize` implementation 
instead of using the derived one from serde. ESTree has some enums whose 
variants are themselves enums, for example we have 

```rust 

enum ModuleItem { 

ImportExportDeclaration(ImportExportDeclaration), 

Statement(Statement), 

} 

enum ImportExportDeclaration { ... } 

enum Statement { ... } 

``` 

This sort of works with serde's derive implementation: you have to use the "tag" 
representation for the inner enums, and an "untagged" representation for the 
outer one (ModuleItem). The problem is that with an untagged representation, 
serde doesn't know what type of data it's expecting. All it can do is go one by 
one and try to parse the data as the first variant (eg ImportExportDeclaration) 
then the next one (Statement) and fail when it gets to the end of the list. If 
the data isn't valid for any reason, deserialization will fail with a "not a 
valid ModuleItem" error. That's true but not helpful, especially if you're 
developing estree, are confident that the input json is valid, and need to 
figure out where you messed up the definition. It's also not helpful as an 
end-user if you're not sure your input json is valid. 

So this PR updates our codegen to emit a custom derive implementation that is 
identical for both regular enums (like Statement) and recursive ones (like 
ModuleItem). We first extract the tag to know what type the value is, then 
deserialize exactly as that type. So in the above case, rather than have to 
first try parsing every ModuleItem as an ImportExportDeclaration and then fall 
through to statement, we just decode the tag (`type` in our case, for example 
say it's an "ForStatement"), then deserialize directly as that type (eg, as 
ForStatement), then wrap it in the enum variant (ModuleItem::Statement(...)). 
For recursive enums like ModuleItem we add an extra wrapper as necessary. 

The end result is that we get much more precise errors and deserialization is 
more efficient: we always decode just the tag, then as exactly that type. 

Note that our serialization is also not perfect right now, because we don't 
always emit the `type` key. Serde only emits it when a value appears in an enum. 
We can similarly generate custom serializers for all our types to always emit 
the tag. That will be straightforward when it's necessary. The current PR was 
more of a blocker, because it was really hard to figure out mistakes in the 
estree definition given the ambiguous errors. Thanks to this PR we now get 
precise errors along the lines of "unknown type `JSXElement`" which are easy to 
resolve.

compiler/forget/crates/estree-codegen/src/codegen.rs

@@ -33,11 +33,6 @@ impl Grammar {
             operators,
         } = self;
 
-        let enum_names: HashSet<String> = enums.keys().cloned().collect();
-
-        let mut node_names: Vec<_> = nodes.keys().cloned().collect();
-        node_names.sort();
-
         let objects: Vec<_> = objects
             .iter()
             .map(|(name, object)| object.codegen(name))
@@ -48,7 +43,7 @@ impl Grammar {
             .collect();
         let enums: Vec<_> = enums
             .iter()
-            .map(|(name, enum_)| enum_.codegen(name, &enum_names))
+            .map(|(name, enum_)| enum_.codegen(name, &enums))
             .collect();
         let operators: Vec<_> = operators
             .iter()
@@ -223,7 +218,7 @@ pub struct Enum {
 }
 
 impl Enum {
-    pub fn codegen(&self, name: &str, enums: &HashSet<String>) -> TokenStream {
+    pub fn codegen(&self, name: &str, enums: &IndexMap<String, Enum>) -> TokenStream {
         let mut sorted_variants: Vec<_> = self.variants.iter().collect();
         sorted_variants.sort();
 
@@ -232,7 +227,7 @@ impl Enum {
             .iter()
             .map(|name| {
                 let variant = format_ident!("{}", name);
-                if enums.contains(*name) {
+                if enums.contains_key(*name) {
                     quote!(#variant(#variant))
                 } else {
                     quote!(#variant(Box<#variant>))
@@ -245,7 +240,7 @@ impl Enum {
                 #(#variants),*
             }
         };
-        let enum_ = if sorted_variants.iter().any(|name| enums.contains(*name)) {
+        let enum_ = if sorted_variants.iter().any(|name| enums.contains_key(*name)) {
             // contains recursive enum, use untagged serialization
             quote! {
                 #[serde(untagged)]
@@ -258,9 +253,92 @@ impl Enum {
             }
         };
 
+        let enum_tag = format_ident!("__{}Tag", name);
+        let mut seen = HashSet::new();
+
+        // tag_variants is used to generate an enum of all the possible type tags (`type` values)
+        // that can appear in this enum. we emit this enum and derive a deserializer for it so that
+        // our enum deserializer can first decode the tag in order to know how to decode the data
+        let mut tag_variants = Vec::new();
+
+        // once the tag is decoded, we need to match against it and deserialize according the tag (`type`)
+        // tag_matches are the match arms for each type.
+        let mut tag_matches = Vec::new();
+
+        // Imagine a case like:
+        // enum ModuleItem {
+        //   ImportDeclaration, // struct
+        //   Statement // another enum
+        // }
+        // We need to generate matches for all the possible *concrete* `type` values, which means
+        // we have to expand nested enums such as `Statement`
+        for variant in self.variants.iter() {
+            if let Some(nested_enum) = enums.get(variant) {
+                let outer_variant = format_ident!("{}", variant);
+                for variant in nested_enum.variants.iter() {
+                    // Skip variants that appear in multiple nested enums, we deserialize
+                    // as the first listed outer variant
+                    if !seen.insert(variant.to_string()) {
+                        continue;
+                    }
+                    // Modeling ESTree only requires a single level of nested enums,
+                    // so that's all we support. Though in theory we could support arbitrary nesting,
+                    // since ultimately we're matching based on the final concrete types.
+                    assert!(!enums.contains_key(variant));
+
+                    let inner_variant = format_ident!("{}", variant);
+                    tag_variants.push(quote!(#inner_variant));
+
+                    tag_matches.push(quote! {
+                        #enum_tag::#inner_variant => {
+                            let node: Box<#inner_variant> = <Box<#inner_variant> as Deserialize>::deserialize(
+                                serde::__private::de::ContentDeserializer::<D::Error>::new(tagged.1),
+                            )?;
+                            Ok(#name::#outer_variant(#outer_variant::#inner_variant(node)))
+                        }
+                    });
+                }
+            } else {
+                if !seen.insert(variant.to_string()) {
+                    panic!(
+                        "Concrete variant {} was already added by a nested enum",
+                        variant
+                    );
+                }
+                let variant_name = format_ident!("{}", variant);
+                tag_variants.push(quote!(#variant_name));
+
+                tag_matches.push(quote! {
+                    #enum_tag::#variant_name => {
+                        let node: Box<#variant_name> = <Box<#variant_name> as Deserialize>::deserialize(
+                            serde::__private::de::ContentDeserializer::<D::Error>::new(tagged.1),
+                        )?;
+                        Ok(#name::#variant_name(node))
+                    }
+                })
+            }
+        }
         quote! {
-            #[derive(Serialize, Deserialize, Clone, Debug)]
+            #[derive(Serialize, Clone, Debug)]
             #enum_
+
+            #[derive(Deserialize, Debug)]
+            enum #enum_tag {
+                #(#tag_variants),*
+            }
+
+            impl <'de> serde::Deserialize<'de> for #name {
+                fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
+                where D: serde::Deserializer<'de> {
+                    let tagged = serde::Deserializer::deserialize_any(
+                        deserializer,
+                        serde::__private::de::TaggedContentVisitor::<#enum_tag>::new("type", "Pattern")
+                    )?;
+                    match tagged.0 {
+                        #(#tag_matches),*
+                    }
+                }
+            }
         }
     }
 }