Issue 639 Check diffs between TUP and IUP

[macchiati]

Add a test of the differences as per #639

[macchiati]

Raw data for #639 is at https://docs.google.com/spreadsheets/d/1mFpC03q_dBNXuVRA_qJ_tHSjmVQY8dIDC4PYRkZW1KA

Property	Analysis (so far)
Bidi_Mirroring_Glyph	IUP is returning null if there is none, TUP returning the code point
Bidi_Paired_Bracket	TUP appears wrong, returning \x{0} character.
FC_NFKC_Closure	IUP is returning the default (equal code point)
Joining_Group	Investigate single difference: IUP Hamza_On_Heh_Goal ≠ TUP Teh_Marbuta_Goal
Joining_Type	2181 differences: IUP Non_Joining ≠ TUP Transparent
Name	IUP is the raw name, TUP is the massaged name, eg <control-0000>
Name_Alias	Apparently TUP doesn't support: IUP NUL; NULL ≠ TUP {EMPTY}
Numeric_Value	Investigate further
Script_Extensions	IUP is "resolved" (layering on top of Script), TUP is raw
ISO_Comment	IUP {NULL} ≠ TUP {EMPTY}
Jamo_Short_Name	just for [ᄋ], IUP {NULL} ≠ TUP {EMPTY}
Unicode_1_Name	IUP {NULL} ≠ TUP {EMPTY}
Numeric_Value	IUP {NULL} ≠ TUP {EMPTY}
*TUP properties missing from IUP, but not important for tests/file generation:	ASCII Case_Fold_Turkish_I IdnOutput Non_Break cjkAccountingNumeric cjkCompatibilityVariant cjkIICore cjkIRG_GSource cjkIRG_HSource cjkIRG_JSource cjkIRG_KPSource cjkIRG_KSource cjkIRG_MSource cjkIRG_SSource cjkIRG_TSource cjkIRG_UKSource cjkIRG_USource cjkIRG_VSource cjkOtherNumeric cjkPrimaryNumeric cjkRSUnicode isNFC isNFD isNFKC isNFKD toNFC toNFD toNFKC toNFKD

[macchiati]

worked out the kinks, and got normalization to use the newer parsed data. Would like to merge this before continuing on.

[eggrobin]

This could use a comment explaining what it is for (as far as I can tell it makes an IUP behave like TUP for testing/diffing purposes, but we shouldn’t actually be used beyond that, right?).

[macchiati]

done

[eggrobin]

Why not just NormalizationForm?

[eggrobin]

IUP is the raw name, TUP is the massaged name, eg <control-0000>

From my experiments with #502, it also looks like IUP has # rather than the code point for CJKV ideographs etc.

[macchiati]

IUP is the raw name, TUP is the massaged name, eg <control-0000>

From my experiments with #502, it also looks like IUP has # rather than the code point for CJKV ideographs etc.

Yes, it uses the same convention for names that the XML does for names. There is a method that resolves that (IndexUnicodeProperties.getName(), also a method that gets a sequence of names for a string.

We could rethink the property to have the fully resolved name if we want. The only reason for the # is just to avoid the storage cost, but that may not be an issue now.