UnicodeSet Properties from PPUCD (L3a) - Binary Properties

[echeran]

This issue represents the first part ("L3a") of the functionality we want from instantiating UnicodeSets that represent Unicode Properties (#168 - "L3").

Starting from the L3a design doc that that @EvanJP created, this issue represents supporting only a more well-defined subset of user input than in full L3 functionality. For a specified short list of binary and enum properties, whose data is collected and regularized in the Pre-Parsed Unicode Character Database (PPUCD), we want to create a separate static constructor-like function that can create the UnicodeSet for that property.

[sffc]

Discussion from 2020-09-03:

Functions or Enums?

• @mihnita: Can we use an enum instead of a function?
• @sffc: I prefer functions because (1) programmers know at build time which properties they want to query, so by having different functions, we can do code slicing; and (2) different properties have different return types (UnicodeSet or UCPTrie).
• @mihnita: At runtime, programmers might not know whether a character is lowercase or uppercase, for example. They will want to loop over multiple properties to test.
• @EvanJP: Could we have both?
• @sffc: For the uppercase/lowercase use, we can have multiple functions: one for getting the set of lowercase characters via UnicodeSet ([:Lu:]), and one for getting the type of letter via UCPTrie.
• @sffc: It sounds like this is an ergonomic versus functional API argument.
• @echeran: We should also have the pattern parsing for the more sophisticated operations. But we should still have the individual functions for code slicing purposes.
• @sffc: Since the functional API (individual functions) is the building block, we should start with that.
• @mihnita: OK

Code Generation?

• @sffc: I think manual would be OK because (1) properties don't get added that often, and when they do, we can have a human check for consistency; and (2) because we may want to write our own documentation for each individual function, and code generation may not be able to generate docs.
• @mihnita: Manual maintenance happens so rarely that I'm not convinced it's worth the trouble to automate.
• @EvanJP: The concern I have is that if we manually generate these functions, how do we test them? With code generation, we could also generate tests.
• @sffc: The rustdoc tests could have a positive example and a negative example, and that might be sufficient test coverage.
• @EvanJP: Properties are complex, and I feel that it would be useful to generate more robust tests.
• @sffc: For data-driven testing, we could implement that on top of L3b.
• @sffc: Code generation is messy. ICU has a lot of it. If we have code generation, we shouldn't check in the artifacts.
• @zbraniecki: For Rust code generation, you can have a "cargo regenerate data". And since code generation is in Rust, dependencies are easier to resolve. So code generation in Rust is a bit nicer than C++. But I can't comment on long-term maintainability.

Next steps

• @EvanJP: I can take a shot at it this weekend. I need to read more about the ppucd file.
• @sffc: A good first step would be generating an inversion list for your favorite binary property from ppucd.
• @echeran: I'm happy to help you out with this too.

[echeran]

I finished converting the L3 design doc into an L3a-specific design doc, at the same place:
https://docs.google.com/document/d/10z0RK7WC7pVOIP9fCZQfcGKCoMvsOcXZIpr86UN9M1Q/edit#

I hope that captures the scope of what we're currently considering, and what are the main tradeoffs. I've marked off which are questions with alternate options, even if some might have more obvious conclusions than others.