Error contexts and the permutation combinator

[nickpiaddo]

Hi,

First of all, thank you for winnow. I am fairly new to Rust, and working with
winnow is an absolute delight.

Context

I made a minimal program to illustrate my problem. See the files in
example.tar.gz:

.
├── Cargo.lock
├── Cargo.toml
└── src
    ├── main.rs
    ├── parsers.rs
    └── sample.nix

Description

I am trying to parse the following structure in a nix file:

settings = {
  # Primary boot-key.
  primary = {
      id = "usb-JetFlash_Transcend_16GB_89J2FGNY7QARQPTD-0:0-part1";
      label = "primary";
      creationDate = "2024-10-29T10:20:00+00:00";
      lastUpdated = "2024-10-29T15:20:00+00:00";
    };

  # List of backup boot-keys.
  backups = [];

  # Maximum number of decryption attempts before program termination
  # range [1-65535].
  maxAttempts = 3;

  # Mount location of the primary boot-key.
  mountPoint = "/boot";

  # Refuse to boot from a bootkey other than the primary or a member
  # of the backup list if set to 'true'.
  checkBootkeyIntegrity = false;

  # Location of the archive containing a LUKS keyfile and header file
  # necessary for device decryption.
  decryptionArchive = "/etc/nixos/initrd/secrets/secboot";
};

settings is an attribute-set with attributes primary, backups,
maxAttempts, mountPoint, checkBootkeyIntegrity, and decryptionArchive.

I am using a function to parse settings resembling the snippet below (taken
from the parsers.rsfile attached to this question).

It is a streamlined version of the one I am developing. Attributes can come in
any order. Hence, an application of the permutation combinator.

pub fn settings<'i, E>(input: &mut Input<'i>) -> PResult<(), E>
where
    E: ParserError<Input<'i>> + AddContext<Input<'i>, StrContext>,
{
    delimited(
        delimited(ws, literal("settings"), preceded(ws, equal)),
        delimited(
            delimited(ws, open_brace, ws),
            permutation((
                preceded(delimited(ws, skip_comments, ws), primary),
                preceded(delimited(ws, skip_comments, ws), backups),
                preceded(delimited(ws, skip_comments, ws), max_attempts),
                preceded(delimited(ws, skip_comments, ws), mount_point),
                preceded(delimited(ws, skip_comments, ws), check_bootkey_integrity),
                preceded(delimited(ws, skip_comments, ws), decryption_archive),
            ))
            .void(),
            preceded(ws, close_brace),
        ),
        terminated(semi_column, ws),
    )
    .context(StrContext::Label("attribute-set `settings`"))
    .parse_next(input)
}

Each parser for primary, backups, etc. reports a context when an error occurs.

permutation will succeed if all child parsers succeed. Currently, if any of
the child parsers fails permutation will always report the context of the last child
parser in the tuple (i.e. decryption_archive).

For example, if I introduce an error in the input, here a £ sign before label:

settings = {
  # Primary boot-key.
  primary = {
      id = "usb-JetFlash_Transcend_16GB_89J2FGNY7QARQPTD-0:0-part1";
      £label = "primary";
      creationDate = "2024-10-29T10:20:00+00:00";
      lastUpdated = "2024-10-29T15:20:00+00:00";
    };

  # List of backup boot-keys.
  backups = [];

  # Maximum number of decryption attempts before program termination
  # range [1-65535].
  maxAttempts = 3;

  # Mount location of the primary boot-key.
  mountPoint = "/boot";

  # Refuse to boot from a bootkey other than the primary or a member
  # of the backup list if set to 'true'.
  checkBootkeyIntegrity = false;

  # Location of the archive containing a LUKS keyfile and header file
  # necessary for device decryption.
  decryptionArchive = "/etc/nixos/initrd/secrets/secboot";
};

The program will report:

error: invalid attribute `decryptionArchive`
  |
2 |   # Primary boot-key.
  |   ^
  |

Even though decryptionArchive is correct, the error says it is not while
returning the wrong error location.

I would have expected the following error:

error: invalid attribute `label`
  |
5 |       £label = "primary";
  |       ^
  |

How can I get the premutation combinator to report the error context of the
parser that was the farthest along, even it ultimately failed, or the error
context of the first of the remaining child parsers that failed when at least
one succeeded?

The tutorial provides an example for managing fall-through error reports with
the alt combinator using error cuts, but I could not get it to work as I would
like with the permutation combinator.

Thanks for your help.

[epage]

You mentioned the solution (cut_err) but left out what how you tried it and what problem you had.

To deal with a bad key inside a permutation, you can do alt(key_values, fail), adding whatever context works for you.

[nickpiaddo]

Hi,

My apologies for the lack of details. The best I can do at the moment is the
following, with a catch-all by wrapping the permutation combinator with an
alt and fail combinator.

pub fn settings<'i, E>(input: &mut Input<'i>) -> PResult<(), E>
where
    E: ParserError<Input<'i>> + AddContext<Input<'i>, StrContext>,
{
    delimited(
        delimited(ws, literal("settings"), preceded(ws, equal)),
        delimited(
            delimited(ws, open_brace, ws),
            alt((
                permutation((
                    preceded(delimited(ws, skip_comments, ws), primary),
                    preceded(delimited(ws, skip_comments, ws), backups),
                    preceded(delimited(ws, skip_comments, ws), max_attempts),
                    preceded(delimited(ws, skip_comments, ws), mount_point),
                    preceded(delimited(ws, skip_comments, ws), check_bootkey_integrity),
                    preceded(delimited(ws, skip_comments, ws), decryption_archive),
                ))
                .void(),
                fail.context(StrContext::Label("attribute"))
                    .context(StrContext::Expected("primary".into()))
                    .context(StrContext::Expected("backups".into()))
                    .context(StrContext::Expected("maxAttempts".into()))
                    .context(StrContext::Expected("mountPoint".into()))
                    .context(StrContext::Expected("checkBootkeyIntegrity".into()))
                    .context(StrContext::Expected("decryptionArchive".into())),
            )),
            preceded(ws, close_brace),
        ),
        terminated(semi_column, ws),
    )
    .context(StrContext::Label("attribute-set `settings`"))
    .parse_next(input)
}

Input:

settings = {
  # Primary boot-key.
  primary = {
      id = "usb-JetFlash_Transcend_16GB_89J2FGNY7QARQPTD-0:0-part1";
      £label = "primary";
      creationDate = "2024-10-29T10:20:00+00:00";
      lastUpdated = "2024-10-29T15:20:00+00:00";
    };

  # List of backup boot-keys.
  backups = [];

  # Maximum number of decryption attempts before program termination
  # range [1-65535].
  maxAttempts = 3;

  # Mount location of the primary boot-key.
  mountPoint = "/boot";

  # Refuse to boot from a bootkey other than the primary or a member
  # of the backup list if set to 'true'.
  checkBootkeyIntegrity = false;

  # Location of the archive containing a LUKS keyfile and header file
  # necessary for device decryption.
  decryptionArchive = "/etc/nixos/initrd/secrets/secboot";
};

Every child parser in the permutation should skip comment lines then try to
parse a specific attribute. Skipping comment lines will always succeed. So,
whichever child parser is used the comment on line 2 # Primary boot-key.
should be consumed.

    preceded(delimited(ws, skip_comments, ws), primary),
    //       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~   ^^^^^^^
    //       Always succeeds in all children   The error occurs here

Running the program with the function above gives the following error:

error: invalid attribute
expected `primary`, `backups`, `maxAttempts`, `mountPoint`, `checkBootkeyIntegrity`, `decryptionArchive`
  |
2 |   # Primary boot-key.
  |   ^
  |

As far as I can tell, the error message tells me where the parser was before
entering the alt combinator, not where the child parser that went the
farthest failed.

Each child parser does not fail equally, some bail earlier than others.

settings = {
  # Primary boot-key.
  primary = {
# ^
# │
# └ all child parsers, except the `primary` parser, fail here (as expected).
      id = "usb-JetFlash_Transcend_16GB_89J2FGNY7QARQPTD-0:0-part1";
      £label = "primary";
    # ^
    # │
    # └ the `primary` parser fails here
      creationDate = "2024-10-29T10:20:00+00:00";
      lastUpdated = "2024-10-29T15:20:00+00:00";
    };

...snip...

I tried introducing cut_errs on the parts on which I want more detailed error messages.

pub fn settings<'i, E>(input: &mut Input<'i>) -> PResult<(), E>
where
    E: ParserError<Input<'i>> + AddContext<Input<'i>, StrContext>,
{
    delimited(
        delimited(ws, literal("settings"), preceded(ws, equal)),
        delimited(
            delimited(ws, open_brace, ws),
            alt((
                permutation((
                    preceded(delimited(ws, skip_comments, ws), cut_err(primary)),
                    preceded(delimited(ws, skip_comments, ws), cut_err(backups)),
                    preceded(delimited(ws, skip_comments, ws), cut_err(max_attempts)),
                    preceded(delimited(ws, skip_comments, ws), cut_err(mount_point)),
                    preceded(
                        delimited(ws, skip_comments, ws),
                        cut_err(check_bootkey_integrity),
                    ),
                    preceded(
                        delimited(ws, skip_comments, ws),
                        cut_err(decryption_archive),
                    ),
                ))
                .void(),
                fail.context(StrContext::Label("attribute"))
                    .context(StrContext::Expected("primary".into()))
                    .context(StrContext::Expected("backups".into()))
                    .context(StrContext::Expected("maxAttempts".into()))
                    .context(StrContext::Expected("mountPoint".into()))
                    .context(StrContext::Expected("checkBootkeyIntegrity".into()))
                    .context(StrContext::Expected("decryptionArchive".into())),
            )),
            preceded(ws, close_brace),
        ),
        terminated(semi_column, ws),
    )
    .context(StrContext::Label("attribute-set `settings`"))
    .parse_next(input)
}

With the same input, I get the right error line...but the wrong context.

error: invalid attribute `creationDate`
  |
5 |       £label = "primary";
  |       ^
  |

Close but, unfortunately, not what I was looking for 😢

error: invalid attribute `label`
  |
5 |       £label = "primary";
  |       ^
  |

Adding cut_errs in a permutation has also a tricky side-effect. If I keep
the same code, but permute for example the positions of attributes
maxAttempts and primary in the input text like this:

settings = {
  # Maximum number of decryption attempts before program termination
  # range [1-65535].
  maxAttempts = 3;

  # List of backup boot-keys.
  backups = [];

  # Primary boot-key.
  primary = {
      id = "usb-JetFlash_Transcend_16GB_89J2FGNY7QARQPTD-0:0-part1";
      £label = "primary";
      creationDate = "2024-10-29T10:20:00+00:00";
      lastUpdated = "2024-10-29T15:20:00+00:00";
    };

  # Mount location of the primary boot-key.
  mountPoint = "/boot";

  # Refuse to boot from a bootkey other than the primary or a member
  # of the backup list if set to 'true'.
  checkBootkeyIntegrity = false;

  # Location of the archive containing a LUKS keyfile and header file
  # necessary for device decryption.
  decryptionArchive = "/etc/nixos/initrd/secrets/secboot";
};

I get:

error: invalid attribute `primary`
  |
4 |   maxAttempts = 3;
  |   ^
  |

...when I would have expected:

error: invalid attribute `label`
   |
12 |       £label = "primary";
   |       ^
   |

Which is due, I suspect, to permutation not trying the parsers after
primary before reporting an error owing to cut_err cutting the program
short

Using cut_errs like below on each child parser in permutation leads pretty
much to the same outcome.

cut_err(preceded(delimited(ws, skip_comments, ws), primary)),

I went through the documentation, which is by the way one of the best I
have seen so far, but I can't figure out how do I get my program to report an
error location that essentially says:

"I went through the parsers in the permutation combinator, and at least one
failed. Among those who failed, here is where an error occurred for the one
that processed the largest piece of the input you gave me"?

Hope this longer exposé helps 😄

Thanks!

[epage]

cut_err should be placed inside of primary, etc on all of the content after they key has been recognized. This means the alt-like behavior only tries another case if the keyword doesn't match and bails out quickly on further errors. Seeing your primary, etc was what I was hoping for from getting more context.

As far as I can tell, the error message tells me where the parser was before
entering the alt combinator, not where the child parser that went the
farthest failed.

It reports the last failure. If you have an alt(something, fail), then fail consumes nothing and reports an error immediately. To report an error on the keyword you can instead do alt(something, (ws, skip_comments, ws, fail).void())

In general, I would actually recommend against using permutation. It is a high level function that, in theory, can make it fast to write code but it also takes away a lot of control and you have to instead workaround its limitations and it will be very slow. alt is a pretty slow operation on its own and permutation will be even worse. This is made worse by the fact that your parsers are repeatedly trying and throwing away (ws, skip_comments, ws) just because a keyword didn't match.

[nickpiaddo]

Hi,
Thanks for the reply. You said

Seeing your primary, etc was what I was hoping for from getting more context.

Are you saying that you would like to see the whole program? Did the file I attached to my first message fail to decompress?

To give as much information as possible, I attached a minimal project containing the whole code to my initial message. I am attaching it again here example.tar.gz.

[epage]

Oh, sorry, I missed that you had a file attached though attached files don't work too well for talking about code. Its easier if all of the relevant samples are posted or if a reproduction repo is linked.

At this point, it was resolved. I was just trying to clarifying what my earlier intention was.

[nickpiaddo]

No worries,...it happens! 😉

I did not want to clutter by posting the almost 400 lines of parsers.rs in the comment box, guess I was wrong. Sorry! 😓

Will keep in mind the correct procedure from now on.