What is needed for streaming/partial/incomplete parsing?

[epage]

How can we improve the state of things to make the experience better?

[epage]

This is a running log of what has been done

• feat: Auto-detect whether to use complete/streaming parser #28 Auto-detect streaming vs complete
• add streaming benchmarks (#1609) #78
• fix(stream)!: Change from "input" terminology to "stream" #142 Renamed things
• feat(stream)!: Allow runtime selection of the partial/complete #166 Allow runtime changing of partial vs complete
• fix!: Clarify Parser::complete_err is error related #168 Rename Parser::complete to Parser::complete_err to clarify how to deals with complete
• fix(parser)!: Allow literals to be parsers of partial inputs #171 allow "hello" to implicitly be a tag (previously only worked for complete inputs)

[epage]

Background: I've not done much streaming parsing

I suspect there are two main cases:

• Parse just enough for a header which includes a length and then do a complete parsing once the specified data is present
  • We have the length_data combinator for this
• Parse a block until a delimiter (like newlines for ndjson), retrying if there isn't enough data. Once the stream is closed, we know its the true eof and we can act as if the end is complete.
  • We have Parser::complete_err for the final case

[epage]

See also #208 ("Semantics of ErrMode::Incomplete and Needed: need some bytes for what goal?")

[VorpalBlade]

I'm interested in writing a streaming forma-preserving parser for JSON5. E.g. I want to get a stream of events like "begin object", "key", "string value", "newline", "spaces" etc. My goal is to use this for single-pass editing (while minimising the git diff), so I have no need to build up a full document structure like toml_edit does.

I deeply care about speed of this program, I will be processing relatively small files, but lots of them. This leads me to believe I should load the full file before parsing, so that I can borrow from the input where possible. Since pretty much everything will have to be treated as opaque string data anyway (otherwise I might not preserve float formatting correctly) I should be able to borrow almost all data.

(This is at odds with what you would do if you wanted to stream process a very large file (e.g. larger than fits in RAM), where you would want to load parts of the file in small chunks. In that case borrowing from the input buffer would be complicated (as it would be ephemeral).)

I'm wondering if this is something that winnow can currently support, or if I'm better served looking elsewhere. It doesn't seem to fit either of the two use cases you mentioned in #169 (comment)

[epage]

This discussion is focused on when the input you are parsing is streaming in, rather than streaming out tokens.

For the latter, we have iterator or you can more manually incrementally parse with parse_next.