LALRPOP for a grammar like HCL (Terrarform)

[dan-cooke]

I am building an interface definition language, it supports generic data types like Structs

struct MyStruct {
   whatever: String,
   names: [SomeOtherStruct, TestStruct]
}

But it also has many "special" types that only allow users to define fields from an explicit Schema, e.g

service Something {
     version: "1.0.0"
     operations: [Blah]
}

Now I have setup a fairly abstrtact AST so one way of parsing this would be via a generic AST Statement with only id and properties in it.

Statement: Statement = {
	"service" <id:Ident> "{" <properties: Comma<Prop>> "}" => Statement:ServiceDef{ id, properties }
...
}

But this would allow users to define fields that we do not allow,

How should we handle a language with an explicit schema, e.g HashiCorpLanguage or similar

1. My First attempt was to parse the schema in Lalrpop itself

ServiceProps: Vec<Property> = {
  <id:r"[^version]"> ":" <v:r#""[^"]*""#> => v
}

But I get an error

  /home/dan/dev/omni/omni_parser/src/omni.lalrpop:29:7: 29:19 error: terminal `r#"[^version]"#` does not have a pattern defined for it

      <id:r"[^version]"> ":" <v:r#""[^"]*""#> => v

I believe I have to add all these schema properties into the Lexer tokens, which feels a bit wrong as they are not exactly Keywords, just data properties. There also may be quite a lot of them.

2. My second attempt was to handle this schema validation on a second pass after lalrpop has parsed the service

This feels like the most sensible approach, first I define an IntermediateRepresentation enum for each of the explicit schema parts.

pub enum IRStruct {
    ServiceDef {
        id: Identifier,
        properties: Vec<Property>,
    },
    OperationDef {
        id: Identifier,
        properties: Vec<Property>,
    },
}

Then I can handle all the conversions and validation on a second pass using a TryFrom impl

impl TryFrom<IRStruct> for Statement {
    type Error = Error;

    fn try_from(value: IRStruct) -> Result<Self, Self::Error> {
        match value {
            IRStruct::StatementDef { id, properties } => {
                let version = properties.iter().find(|x| x.id.name == "version").map_or(
                    Ok("0.1.0"),
                    |p| match p.value {
                        omni_ast::Expression::Literal(_) => todo!(),
                        _ => {
                            return Err(Error::TypeError(ErrorDetails {
                                message: format!("version must be a string literal"),
                            }))
                        }
                    },
                )?;
                Ok(())
            }
            IRStruct::OperationDef { id, properties } => todo!(),
        }
    }
}

(this is just a quick draft, I think I can probably use a crate for validation to make this conversion less verbose)_

What is the recommended way to handle something like this?

[Pat-Lafon]

Not sure what others would recommend but I would certainly go with the second approach. It is a nice separation of concerns, it keeps your parsing grammar simpler, and it will allow you to have finer grain control on the error messages you return in case validation fails.

[dan-cooke]

yeah I am definetly leaning towards the second option

thanks for your input!

[yannham]

Note that in LALRPOP you have fallible actions. You could potentially avoid (or not, I don't know enough of your use-case to say for sure) the first lax representation altogether and just validate your additional conditions right in the LALRPOP action, either failing or producing a well-formed statement directly.

[dan-cooke]

@yannham that was another solution I tried, but I don’t love writing tonnes of rust code in my LALRPOP files without LSP/ syntax highlighting(at least I can’t get it to work in neovim)

[yannham]

What I usually do is to write most of the code outside of the actual LALRPOP grammar file, in different modules such as ast, error, etc. For little helpers that don't fit anywhere, it could be utils. Anyway, the idea is that LALRPOP action code is mostly just calling to a few existing methods define outside of the grammar, and rarely exceeds 10 lines, which I find manageable in practice.

Typically, you could just call your try_from directly from within the action code. If the things you check are hard constraints of your grammar, IMHO it makes sense that the parser return a Statement or an error, and that the IRStruct isn't really there to be observed but more of an implementation detail. And if you just build an IRStruct to call try_from immediately thereafter, you can probably get rid of the IRStruct altogether ("inlining it" in the arguments of a method).

Sometimes it does make sense to keep the lax IR, for example if you need to maintain a Concrete Syntax Tree for an LSP, or because you have any interest in doing other non trivial operations on this IR. Or maybe inlining the struct makes the arguments list too big, and it's better just as packed way to pass around data required to build Statement. Your mileage may vary! But I just wanted to mention fallible actions in case you've missed them 🙂

[dan-cooke]

@yannham thats exactly what I was planning to do with option 2 - the try_from was defined in a separate module and my LALRPOP action is super light.

some great points about keeping the lax ir too - I definetly want to support LSP with treesitter so something to consider

thank you!