Data schema for ICU4X

[sffc]

I've so far left the question of the ICU4X data schema a bit undefined. I wanted to discuss this in more detail now.

I was thinking of something along the lines of /category/key/[payload/]locale, followed by a metadata block that includes the hunk. Potential new names that are more domain-specific: /category/subcategory/[flavor/]locale.

Why feature-first and not locale-first?

• Different features can have different locales.
• Keeping the locale near the leaf of the tree means that you don't have to re-crawl the tree in order to perform fallbacks.

For example:

{
  "plurals": {
    "cardinal_v1": {
      "en": {
        "data": {
          "one": "..."
        }
      },
      "ru": {
        "data": {
          "one": "...",
          "few": "...".
          // ...
        }
      }
    },
    "ordinal_v1": {
      // ...
    }
  },
  "numbers": {
    "currency_symbols_v1": {
      "EUR": {
        "sr": {
          // sr implicitly inherits from und
          "data": { /* ... */ }
        },
        "sr_Latn": {
          // sr_Latn should inherit from und, not sr
          "fallback": "und",
          "data": { /* ... */ }
        },
        "und": {
          "data": { /* ... */ }
        }
      },
      "GBP": { /* ... */ }
    }
  }
}

This structure can optionally be broken down onto the filesystem in order to get more manageable file sizes. The filesystem structure can also be used to serve this as static data from a HTTP server.

📦data
 ┣ 📂numbers
 ┃ ┗ 📂currency_symbols_v1
 ┃   ┣ 📂GBP
 ┃   ┃ ┣ 📜sr.json
 ┃   ┃ ┣ 📜sr_Latn.json
 ┃   ┃ ┗ 📜und.json
 ┃   ┗ 📂USD
 ┃     ┗ 📜und.json
 ┗ 📂plurals
   ┣ 📂cardinal
   ┃ ┣ 📜en.json
   ┃ ┗ 📜ru.json
   ┗ 📂ordinal
     ┣ 📜en.json
     ┗ 📜ru.json

To generate this schema, we will need a tool that transforms from CLDR (either XML or JSON). Ideally that tool would be written in Rust, as it could depend on ICU4X, e.g. to process data into a more convenient form.

@zbraniecki @nciric

[sffc]

We discussed this on 2020-07-17 but did not reach a final conclusion. We will continue async. CC @mihnita

[sffc]

The layout I proposed in the OP is being implemented in #200. I filed #211 to follow up with an alternative data provider using a language pack filesystem layout. @mihnita @nciric