RFC: Turborepo CLI BYOK Artifact Integrity and Authenticity Verification

[gaspar09]

Motivation

Turborepo CLI caches build artifacts locally. When linked to a remote cache, Turborepo CLI will upload those artifacts so that they can be downloaded by teammates. The contents of artifacts can range from compiled code to event logs emitted during build. Artifacts uploaded to the remote cache are downloaded and used in local builds or CI builds.

We would like to provide a mechanism to verify integrity and authenticity of the downloaded artifacts: that the artifacts are genuine and have not changed since upload.

Terminologies Used in this RFC

• BYOK: Bring your own keys, The secret key is provided by the client using Turborepo

Goals

• Turborepo CLI signs the artifacts with a secret key before they are uploaded to the remote cache.
• Turborepo verifies the signature integrity and authenticity of artifacts before they are applied.
• BYOK: Users of Turborepo CLI can provide their own secret keys that are used for Signing and Verifying.

Non-Goals

• Turborepo CLI encrypts the artifact
• The Remote Cache will support Key Rotations
• Figure out how to do secret key distribution for teams

Threat Model

Artifacts are blobs of data that are sent to the Remote Cache. These artifacts can be scoped to an individual user or team. If they are scoped to team, then any members of the team have access to download artifacts uploaded by their teammates. For the purposes of this discussion we will assume that artifacts are scoped to a team. In general we can replace team ID with user ID if the scope is for users. Artifacts can not change scopes once uploaded.

The artifact uploads and downloads use user Authentication to the Remote Cache . The artifacts are uploaded and downloaded using the user token, artifact hash, and team ID via PUT and GET requests to /v8/artifacts/:hash?teamId={teamId}, respectively.

Overview of Security Goals:

We would like to guarantee that a user of Turborepo CLI can assert the integrity and authenticity of a downloaded artifact. Any artifact that is downloaded using GET /v8/artifacts/:hash?teamId={teamId} is exactly the same artifact that is uploaded to PUT /v8/artifacts/:hash?teamId={teamId}.

Enumeration of Attacks focusing on Code Injection:

• Storage tampering attack: The attacker modifies the Remote Cache artifact.
• Mode switching attack: The attacker disables artifact signing on a compromised user’s Turborepo CLI config.
• Replay attack: The attacker replays a Remote Cache artifact tag on PUT or GET request for a given hash and teamId.
• Input Manipulation Attack: A network attacker manipulates the artifact hash, teamId, or signature when the artifact is uploaded.

Security Non-Goals:

• We are not defending against the exfiltration of artifacts. Examples include:
  • Network attacks that redirect artifact uploads or downloads to other entities.
  • A compromised remote cache exfiltrates artifacts to an adversary.
• We are not defending against attacks that may destroy artifacts either by manipulation so that they are rejected by Turborepo CLI or that they are erased so they are not available.

Proposal

Option 1: Single Public-Private Key pair signatures by a key manager service

NOTE: Not specified for this option - Changes to turbo.json config or changes to headers in Turborepo CLI PUT / GET requests.
See other options for related discussion.

We assume that Turborepo CLI is authenticated to request asymmetric key signing and verification from an external key manager. We assume Turborepo CLI has access to the public key but not the private key.

graph TD;

    K[KeyManagerService]
		V{Verify?} 
    S[User 1]
    C[(Remote Cache)]
    R2[User 2]
    AD[\artifact\]

    S-->|"artifact, signature"|C
    S-->|"SHA256(artifact)"|K
    K-.->|signature|S
    C --> |"artifact, signature"|R2
    R2 -.-> |"artifact, signature"|V

		
    V -.->|verify signature locally <br/> with public key |V
    V -.-> |verified| AD
    V -.-> |not verifed|Reject

Loading

Locally, Turborepo CLI does not have access to the private key to sign the SHA256(artifact) digest. Turborepo CLI will request the signature from the Key Manager Service and forward the signature and artifact to the Remote Cache. The receiving user will download the artifact and signature. Turborepo CLI will verify the signature for SHA256(artifact) using the public key.

There is only a single Public-Private key pair that is used for verifying and creating signatures. Thus, this option doesn't provide a mechanism through Turborepo CLI to revoke Signing access to specific users. The key manager should be responsible for authenticating users` signature requests and revoking access when necessary.

The key manager should not share the private key with individual users. Signatures by the key managers are proof that an authenticated user generated the artifact.

Option 2: Per User Public-Private Key Signatures

We assume that Turborepo CLI is using a strong Public-Private key pairs associated with individual users and certified by the Team. Teams using Turborepo will be responsible for establishing the authenticity of Public Keys.

Turborepo CLI will computes the following for each artifact and append the values as headers to the Remote Cache uploads.

• x-artifact-digest
  • The SHA256 over the artifact and JSON stringified metadata (hash, teamId, userId and artifact expiration)
• x-artifact-signature
  • The signature generated over the x-artifact-digest
• x-artifact-signature-metadata
  • The stringified metadata used to associate the signature with a user

Enable Signing on Turborepo CLI

Turborepo will specify an enableSignatureVerification flag on the turbo.json config.

// turbo.json
{
// turbo config
...

// When this flag is present, all uploaded and downloaded artifacts
// are expected to be signed with an RSA or ECDSA signature. 
"enableSignatureVerification": true,
// minutes that an artifact hash is valid
"expiration": 86400, 

{

Turborepo CLI BYOK: Key Format and Retrieving the Public Private Keys

Turborepo CLI requires:

• A per user, a Public-Private Key pair. (e.g. RSA, ECDSA)
• The list Public Keys for all users on the team indexed by userId or email identifier similar to PGP

We assume that the team has an authenticated method to distribute public-private key pairs via some certificate system or similar.

Turborepo CLI can follow a pattern like viper and support decreasing precedence lookups for the user public-private key pair and the list of teammates public keys. Alternatively, these locations can be set and enforced on the turbo.json config.

Turborepo CLI signs artifacts and verifies

We will refer to the user that generates the artifact as the sender, and the user that downloads the artifact as the receiver. Assume the sender and receiver have trusted public-private key pairs.

==== Sender Steps ====

1. Sender with userId: sender generates artifact with metadata
   1. metadata is the JSON.stringify({hash, teamId, userId: "sender", expiration})
2. Turborepo CLI computes the sha256 artifact digest from SHA256(metadata||artifact) and signs it using the senders private key to generate signature_by_sender(digest).
3. Turborepo CLI uploads the artifact to the remote cache and sends the following headers
   1. x-artifact-digest : digest
   2. x-artifact-signature: signature_by_sender(digest)
   3. x-artifact-signature-metadata: metadata

===== Receiver Steps ====

4. The Receiver downloads the artifact and relevant object metadata from the remote cache, which includes digest , signature_by_sender(digest), and metadata
5. Turborepo CLI computes the digest_new from SHA256(metadata||artifact) and compares it to the digest received from the download.
   1. If digest_new != digest then Turborepo CLI throws an error.
6. Otherwise, Turborepo CLI looks up the public key for sender because that is the userId specified in the metadata.
7. Turborepo CLI verifies the signature_by_sender(digest) using digest and senders public key. If the verification fails Turborepo CLI throws an error.

Defense against enumerated attacks

• Storage tampering attack: The attacker modifies the Remote Cache artifact.
  • Verifying the x-artifact-signature on the downloaded artifact mitigates this attack by asserting that the downloaded artifact is unchanged.
• Mode switching attack: The attacker disables signing on a compromised user’s Turborepo CLI config.
  • Turborepo requires verifying downloaded artifacts when enableSignatureVerification is true or when a downloaded artifact has a x-artifact-signature header. Thus, any unverified artifact will not be applied by Turborepo CLI.
• Replay attack: The attacker replays a Remote Cache artifact and Signature tag on PUT or GET request for a given hash and teamId.
  • Every uploaded artifact has an authenticated expiration field. Turborepo CLI will verify this expiration against the current time to verify downloaded artifacts are still valid. This expiration is configured in turbo.json and can be on the order of a few days or tuned based on artifact usage frequency.
• Input Manipulation Attack: A network attacker manipulates the artifact hash, teamId or signature when the artifact is uploaded.
  • The x-artifact-signature authenticates the hash , teamId . Thus, Turborepo CLI would reject downloaded artifacts that have these tampered fields.

Option 3: HMAC-SHA256

We assume that Turborepo CLI is using a strong secret key generated by and shared among the team. Turborepo is written in GO, and there is a standard crypto library implementation of this algorithm that can be used to sign and verify artifacts.

The impact here is that the symmetric key used to sign and verify the artifacts will need to be distributed to teammates using Turborepo CLI.

Turborepo CLI will computes the following for each artifact and append the values as headers to the Remote Cache uploads.

• x-artifact-hmac
  • The HMAC-SHA256 taken over the artifact, hash, teamId, hmacExpiration, and hmacKeyVersion
• x-artifact-hmac-key-version
  • The version id of the hmac key to use to compute the x-artifact-hmac
• x-artifact-hmac-expiration
  • The expiration date of the x-artifact-hmac to prevent replaying old artifacts downloads

Enable Signing on Turborepo CLI

Turborepo will specify an enableHMAC flag on the turbo.json config.

// turbo.json
{
// turbo config
...

// When this flag is present, all uploaded and downloaded artifacts
// are expected to be signed with an HMAC tag. 
"enableHMAC": true,
// The hmacKeyVersion is the current key version that should be used 
// to sign and verify artifact uploads and downloads
"hmacKeyVersion": "1.1",
// minutes that an artifact hash is valid
"hmacTTL": 86400, 

{

Turborepo CLI Accepts User provided secret keys

Getting the Key

Yet to be determined what will be easiest for users to expose the secret key from their machine to the Turborepo CLI. Some options include:

1. turbo.json path to local file config where keys are stored
2. turbo.json configured environment variable where key is accessible
3. Turborepo CLI accepts the key as a command line input on each run
4. Turborepo CLI asks for key once and stores it on global config store on the user machine.

Turborepo CLI can follow a pattern like viper and support decreasing precedence lookups for the symmetric key.

Key Format

Turborepo CLI will version secret keys used in signing. This version will be stored on the turbo.json config. The version is semver style major.minor. Note that the secret key is not stored on the turbo.json config, only the version. This version can be attached to metadata on the uploaded artifact. This is useful for processing key rotations and signaling to users that they’re using an expired secret key.

Turborepo CLI handles Verification

If an artifact is downloaded and the secret-key version on the artifact metadata is a version used sign the artifact, then it must successfully verify. Otherwise, Turborepo CLI should throw an error.

If the secret key version of a downloaded artifact is more recent than the version on the turbo.json config than an error is thrown with a message indicating they don’t have most recent key.

If the secret key version of a downloaded artifact is older than the most recent version of the key on the turbo.json we can either:

• Major version update: Ignore the artifact even if the user has the older versioned secret key.
• Minor version update: Verify the artifact if the user has the older versioned secret key.

Defense against enumerated attacks

• Storage tampering attack: The attacker modifies the Remote Cache artifact.
  • Verifying the x-artifact-hmac on the downloaded artifact mitigates this attack by asserting that the downloaded artifact is unchanged.
• Mode switching attack: The attacker disables HMAC tagging on a compromised user’s Turborepo CLI config.
  • Turborepo requires verifying downloaded artifacts when enableHMAC is true or when a downloaded artifact has a x-artifact-hmac header. Thus, any unverified artifact will not be applied by Turborepo CLI.
• Replay attack: The attacker replays a Remote Cache artifact and HMAC tag on PUT or GET request for a given hash and teamId.
  • Every uploaded artifact has an HMAC authenticated expiration. Turborepo CLI will verify this expiration against the current time to verify downloaded artifacts are still valid. This expiration is configured in turbo.json and can be on the order of a few days or tuned based on artifact usage frequency.
• Input Manipulation Attack: A network attacker manipulates the artifact hash, teamId, hmacExpiration,hmacKeyVersion or HMAC tag when the artifact is uploaded.
  • The x-artifact-hmac authenticates the hash , teamId , hmacExpiration and hmacKeyVersion. Thus Turborepo CLI would reject downloaded artifacts that have these tampered fields.

Turborepo CLI pseudocode

const crypto = require("crypto");

/**
 * Required inputs to Turborepo.
 */
//  Secret generated by customer. Not shared with Remote Cache
const keyLength = 256 / 8; // 256 bit in bytes
const key = crypto.generateKeySync("hmac", { length: keyLength });
// Used by turborepo CLI to identify the key that should be used to verify the artifact
const hmacKeyVersion = "1.1";
// Team ID
const teamId = "team_xxx";
// User token generated using `turbo login`
const token = "user_token";

/**
 * Required fields computed by turborepo
 */
// tar binary file to be uploaded to Remote Cache
const artifact = Buffer.from("artifact contents");
// Turborepo derived hash for `artifact`
const hash = "xxx";

/**
 * Generate HMAC tag for artifact, using customer provided key for the 
 * specified version.
 * Subtle, but taking the hmac over the teamId and hash prevents code 
 * injection by ensuring that the artifact is associated with the hash and team.
 * Otherwise an adversary could replace the artifact and hmac for another 
 * hash on the server. 
 */
async function computeHMACTag(teamId, artifact, hash, hmacExpiration, hmacKeyVersion) {
  const key = await keyForVersion(hmacKeyVersion);
  const metadata = JSON.stringify({teamId, hash, hmacExpiration, hmacKeyVersion});
  return crypto
    .createHmac("sha256", key)
    .update(metadata)
    .update(artifact)
    .digest("hex");
}

/**
 * Turborepo CLI verifies the HMAC key version is allowed and verifies the hmac 
 * tag on the artifact matches the expected HMAC retrieved from the download.
 */
async function verifyArtifactHMAC(
  teamId,
  artifact,
  hash,
  hmacExpiration,
  hmacKeyVersion,
  expectedHMAC
) {
  await verifyHMACKeyVersionIsAllowed(hmacKeyVersion);
  const tag = await computeHMACTag(teamId, artifact, hash, hmacExpiration, hmacKeyVersion);
  if (tag !== expectedHMAC) {
    throw Error("HMAC mismatch");
  } 
  if (hmacExpiration < Date.now()) {
    throw Error("HMAC expiration");
  }
}

/**
 * Turborepo CLI computes HMAC and uploads artifact to remote cache
 */
async function uploadArtifact(teamId, artifact, hash, token) {
  const hmacKeyVersion = await readVersionFromConfig()
  const hmacExpiration = getHmacExpiration() // 3 days from now or some config
  const tag = computeHMACTag(teamId, artifact, hash, hmacExpiration, hmacKeyVersion);

  await fetch(`https://vercel.com/api/v8/artifacts/${hash}?teamId=${teamId}`, {
    method: "PUT",
    headers: {
      Authorization: "Bearer " + token,
      "x-artifact-hmac": `${tag}`,
      "x-artifact-hmac-key-version": `${hmacKeyVersion}`,
	    "x-artifact-hmac-expiration": `${hmacExpiration}`
      // Other headers not relevant to hmac
      "Content-Type": "application/octet-stream",
      "x-artifact-duration": 500, // some task duration metadata
    },
    body: artifact,
  });
}

/**
 * Turborepo CLI downloads and verifies artifacts
 */
async function downloadArtifact(teamId, hash, token) {
  const res = await fetch(
    `https://vercel.com/api/v8/artifacts/${hash}?teamId=${teamId}`,
    {
      method: "GET",
      headers: {
        Authorization: "Bearer " + token
      },
    }
  );

  hmac = res.headers["x-artifact-hmac"];
  hmacKeyVersion = res.headers["x-artifact-hmac-key-version"];
  hmacExpiration = res.headers["x-artifact-hmac-expiration"]; 
  const artifact = res.body;
  await verifyArtifactHMAC(teamId, artifact, hash, hmacExpiration, hmacKeyVersion, hmac);
  // All is good. apply artifact
}

[jaredpalmer]

This has been shipped in turbo 1.2. See the documentation here: https://turborepo.org/docs/features/remote-caching#artifact-integrity-and-authenticity-verification