RFC: Repository metadata reorganization

[lberrymage]

Problem

Accrescent's repository metadata format and organization was not designed for cacheability, internationalization, or atomicity. As a result, Accrescent has limited scalability, target audience, and robustness. These obstacles must be overcome for Accrescent to become a viable alternative to existing app stores with comparable features and a scalable foundation to build on.

Purpose and scope

This RFC aims to thoroughly define the Accrescent repository metadata format(s), directory tree structure, and caching strategies to maximize cacheability, internationalization support, and availability. This design will incorporate both current features of Accrescent's repository metadata and future features which are known to be desired. Undeveloped features need not be fully defined; however, the design must not create any obvious conflicts or challenges for implementing future features.

These proposed changes are being made public to gather feedback on design to ensure the system as implemented meets the needs of Accrescent and won't require significant changes to the repository organization for the foreseeable future. If you have any questions, suggestions, comments, or concerns, please make them in this issue thread.

The repository metadata should be representable exclusively via static files, so it is assumed that no dynamic server calls are required. The decision to use a static file structure for the repository is out-of-scope for this RFC.

Organization

This section describes the proposed organization at a high level.

Directory structure

Below is the proposed directory structure for the repository metadata.

index.sjson
apps
└── {app_id} (e.g. "app.accrescent.client")
    ├── version
    ├── metadata.json
    ├── {locale} (e.g. "en-US")
    │   ├── listing.json
    │   └── icon.png
    └── {version_code} (e.g. "13085")
        ├── compat.json
        ├── metadata.json
        ├── base.apk
        └── *.apk

File contents

This section describes the contents of each file in the above directory structure.

• index.sjson: A (not quite) JSON file containing the index of the repository's apps along with critical security information.

  • version
    This field represents the current version of the signed repository index. It is monotonically increased on every update to the index contents and is intended as a downgrade protection measure used in tandem with the index signature. A client will refuse accepting the index if its version is less than the latest version it has already validated.
  • apps
    This field represents a map of app IDs currently published in the store to their metadata.
    • min_version_code
      This field represents the minimum version code of this app the client will attempt to install. If the client downloads this app and its version code is less than the respective min_version_code, it will refuse to install it. This is to protect against server-side "downgrades" (maliciouly replacing app files with a legitimate, older version which possibly has security vulnerabilities).
    • signing_cert_hashes
      The list of signing certificate hashes this app must be signed with. Must be non-empty. If it contains more than one item, the app was signed with multiple certificates, and the client will refuse to install the app if the set of certificates the app was signed with do not exactly match those in signing_cert_hashes. If it contains exactly one item, the app was signed with one certificate, and the signing certificate lineage must contain the hash in signing_cert_hashes.
  • signature
    This value contains the signify signature for the rest of index.sjson. The signature is verified by the client before processing any of the rest of the index. Otherwise the index is rejected. The base64 signature is appended directly to end of the otherwise JSON file (so it is not a JSON field) to ensure the signature's cache lifetime is tied to that of the index itself.

• version: A plain text file containing the version code of the latest published version for this app.

• {app_id}/metadata.json: Contains non-version-specific app metadata shown in the UI but not used by the installer.

  • listings
    An array of locale strings representing the available store listing locales.
  • tags
    An array of tags this app is labeled with.

• listing.json: A localized store listing.

  • label
    The label of the app as shown in the store. While app names themselves are usually not translatable, the label may contain translatable strings, for example, "SecureChat: Secure texting," which is why this field is localized.
  • short_description
    A short (up to 80 character) description of the app.
  • long_description
    A long-form (up to 4000 character) description of the app.

• icon.png: A 512x512 square PNG app icon to be displayed in the store listing.

• compat.json: App information used to determine compatability with the current device.

  • min_sdk
    An integer representing the minimum Android SDK version this app supports.
  • abi_splits
    The list of ABIs this app supports through split APKs.
  • lang_splits
    The list of languages this app supports through split APKs.
  • density_splits
    The list of screen densities this app supports through split APKs.
  • required_features
    The list of device features required by this application. Maps directly to required <uses-feature> manifest elements.

• {version_code}/metadata.json: Contains version-specific app metadata shown in the UI but not used by the installer.

  • version_name
    The version name for this version of the app.
  • requested_permissions
    The list of permissions this app may request.
  • changes
    The changelog entry for this version of the app (max 500 characters)

• base.apk: The base APK for this app.

• *.apk: All split APKs for this app.

Caching strategies

The below caching strategies apply both to the Accrescent client and caching proxies.

• version: never, so the client never attempts to fetch stale, nonexistent metadata
• {version_code}/*: infinite, since the metadata for a specific app version never changes
• everything else: finite, since slightly stale data is okay

Usage diagrams

---
title: Index update
---

flowchart LR
    n0((Start)) --> n1(Download index)
    n1 --> n2{Does sig verify?}
    n2 --> |No| n3((Quit))
    n2 --> |Yes| n4{Is version less than current?}
    n4 --> |Yes| n3
    n4 --> |No| n5(Update local index)
    n5 --> n3

Loading

---
title: App Update (for a single app)
---

flowchart LR
    n0((Start)) --> n1
    n1(Download version file) --> n2{Is greater than installed?}
    n2 --> |No| n3((Quit))
    n2 --> |Yes| n4(Download compat info)
    n4 --> n5{Is compatible?}
    n5 --> |No| n3
    n5 --> |Yes| n6[[Download and install update]]
    n6 --> n3

Loading

---
title: Download and install update
---

flowchart LR
    n0((Start)) --> n1
    n1(Download APKs) --> n2{Does sig verify?}
    n2 --> |No| n3((Quit))
    n2 --> |Yes| n4{Is sig identity in repo metadata?}
    n4 --> |No| n3
    n4 --> |Yes| n5{Is newer than min_version_code?}
    n5 --> |No| n3
    n5 --> |Yes| n6(Install app)
    n6 --> n3

Loading

Known issues

• App labels are no longer represented in signed repository metadata, making them vulnerable to modification.
• Icon hashes are no longer represented in signed repository metadata, making icons vulnerable to modification (note that Accrescent doesn't currently implement icon hash verification even though the hashes exist).
   

Unresolved questions

Below are some unresolved questions about how certain metadata should be represented. They don't all need to be addressed before adopting a new metadata organization, but should at least be considered and possibly addressed.

• How should different app channels/tracks (e.g. alpha, beta) be supported?
• How should download statistics be represented?
• How should feature splits / dynamic delivery be supported?
• How should delta updates be supported?
• How should update changelogs be represented?
• How should tags/categories be represented?

[soupslurpr]

Question: How does the way the signing cert hashes get verified not prevent a compromised key that was rotated from being used to update the app? Since it says it must only contain one in the lineage. I know the Android docs say to do this but I'm a little confused and would like an explanation if you know :D

[lberrymage]

Sorry, can you clarify? I can't quite picture the situation you're describing.

[soupslurpr]

An app is signed with a key. Let's say it gets compromised or it just has to be rotated to a newer key that is more secure. If the old key is cracked what's stopping it being put in the signing_cert_hashes and being accepted by the client?

[lberrymage]

Let's say the first certificate is "aaaa" and the second certificate is "bbbb".

The app developer rotates their app's cert from "aaaa" to "bbbb" and uploads a new update to the developer console. The console sees that the cert has been rotated, so it keeps the update from being published temporarily. The repository maintainer (myself) is notified of this, verifies the rotation locally (so the server isn't trusted), and updates signing_cert_hashes for the app from "aaaa" to "bbbb". The index is then signed and uploaded to the repository.

Someone cracks the key for cert "aaaa", gains access to Accrescent's servers, and replaces the legitimate app with a malicious copy signed with cert "aaaa".

When Accrescent downloads the app, it'll see that the app was signed with cert "aaaa". Provided it has an up-to-date repository index, it knows that signing_cert_hashes contains "bbbb", not "aaaa", so it rejects the update as invalid.

Does that answer your question?

[soupslurpr]

Thanks, that answered my question. The wording seems to say that the app certificate must only be contained in the certificate lineage stored in signing_cert_hashes, no?

"If it contains exactly one item, the app was signed with one certificate, and the signing certificate lineage must contain the hash in signing_cert_hashes."

[lberrymage]

The certificate in signing_cert_hashes must exist somewhere in the app's certificate lineage. The full lineage isn't contained in signing_cert_hashes, just the latest cert in the lineage that Accrescent is aware of.

[soupslurpr]

How should update changelogs be represented?

Can't it be in {version_code}/metadata.json or another file in {version_code}? Since it would be specific to the version_code. Is there a reason that won't work?

[lberrymage]

That would make sense if we only want to show the latest changelog entry in the UI. We may want to have the option to view past changelog entries, but I think only being able to view the latest makes sense for now.

[soupslurpr]

If we ever want to view past changelog entries there could just be version code directories in one versions directory or something.

[soupslurpr]

Regarding delta updates, is there anything wrong with just placing the delta for the latest published version in {version_code}/delta or so? I don't think there is a need for storing past deltas?

[lberrymage]

Yes. A delta is tied to two app versions: the one it's updating from and the one it's updating to. Thus, it would be unusable by clients with a different "from" version installed than the "from" version the delta targets.

Storing past deltas may be desirable because not every client can be expected to perform every update. If, say, 100 clients are on v1.0 of an app, 100 are on v2.0, and a new v3.0 update is pushed out, making a delta for only the v2.0 -> v3.0 update still leaves 100 clients performing an update from v1.0 -> v3.0 without a delta.

[soupslurpr]

Maybe it could be a time-based approach? Like storing the deltas for the past week or more of updates to the latest version.