Alternative remote caching hosts

[thib92]

Describe the feature you'd like to request

Turborepo offers remote caching, allowing to share cached artifacts with team members.
Currently two remote caches are available:

• Remote cache hosted by Vercel
• Custom remote cache implemented by the end-user, which mimics the Vercel API

Some users might want to store their remote caches in other object-store-like services, like:

• AWS S3 (or any S3-compatible stored like MinIO)
• Google Cloud Storage
• Azure Blob Storage
• OpenStack Swift
• Anything else that can store files (HDFS, NFS, etc.)

This can be for several reasons. Here are three I can think about:

• I am not using Vercel with my team, and I don't plan to in the forseeable future. I don't want to use yet another cloud provider
• I do not want to share my build artifacts on a product that I can't control the security on (aka I want to manage my S3 security)
• I need to store my artifacts on-prem (aka I want to store this in my own datacenter)

Describe the solution you'd like

Turborepo would offer a configuration option to choose between supported remote cache stores. Users would need to configure access parameters and credentials in the configuration file of Turborepo in their monorepo.
In this case, Turborepo would write and reads its cache from the external cache host configured (S3 bucket for example).

Implicit login would need to be supported too: in the case of S3, Turborepo should be able to use the aws CLI to authenticate its calls directly, without needing an API token / secret key. This is usually supported by the Cloud provider's SDK.

Describe alternatives you've considered

The problem with this is that Turborepo will need to implement code for each supported caching provider. This can lead to a lot of code, and possibly go outside of the scope of the project.

Here are two other solutions I can think about:

A separate proxy for each wanted cache host

Since Turborepo already supports custom HTTP endpoints (that mimic the Vercel API), we could just develop a proxy that will act as the Vercel API, and will just forward operations to the end cache provider (S3, Google Cloud Storage, etc.)

This has the advantage of not bloating the Turborepo main codebase and binary, and allowing users with exotic requirements to push a PR to Turobrepo for a very specific use-case (or forking Turborepo as a last resort).

The downside is that the proxy will have to be hosted somewhere, and that each user will need to host their own version of the proxy.

A plugin-based approach

Another way of doing this would be to delegate the communication with the remote cache to a second binary that lives in the same machine as Turborepo. Turborepo would be configured to delegate all operations to this binary.

This is similar to what Docker does with authentication to alternative registries. Here are examples of documentation: AWS ECR, Google Cloud Container Registry (their standalone helper is the closest I could find to what I'm suggesting)

Turborepo would fork a process of the provider's plugin (aka adapter), and would forward the instructions to it directly.

Again, this has the advantage of not bloating Turborepo's code, and allow for custom implementations without PRing or forking Turborepo.

Terraform also has a plugin based approach, but I'm not sure how they handle it. The Docker model will probably be easier to implement.

[thib92]

Note that it's not all or nothing:

Implementing a few providers does not mean that using a plugin-based or a proxy-based approach is not possible.

Turborepo could implement the most needed providers, and leave the more exotic ones to a plugin-based or proxy-based approach. This allows for most users to only depend on Turborepo, and a few to need an additional moving part.

S3 would probably be the best candidate for an in-tree provider, for two reasons:

1. AWS is the most used Cloud provider
2. The S3 API is a standard that has been re-used by multiple late-comers (MinIO, BackBlaze B2, CloudFlare R2, etc.)

The rest can be implemented by plugins or proxies.

This means that there will be some triaging involved here, as repeated requests for additional in-tree providers will likely come. A fixed list of supported providers with a clear statement that no additional providers will be added to the main repo could reduce this a bit.

I'd recommend a plugin-based approach for the less-used providers, as it is quite simple to install (just have a binary with the right name in your path and you're good to go).

[weyert]

I have written my own remote cache server for turbo yesterday for Google Cloud Storage. I will see if it can be open-sourced

[thib92]

I'm also happy to help with the implementation once the correct solution has been established with the core team. I don't have a lot of experience with Go but that will make for some nice practice!

[m19c]

Yeah, that's a really good idea.
I am also happy to help with the implementation.

@thib92 we can also work together if you wan't to.

[m19c]

One thing the project manager should consider:
Adding different storage providers will also increase the size of the binary.

[shaulgo]

It would be great if there will be an option for a Node custom storage provider. This could be file/files that expose the correct api.

[weyert]

I have talked with my work and I am allowed to open-source my proof of concept server which is compatible with the remote cache of Turborepo and supports Google Cloud Storage, and Amazon S3. Azure Storage is also possible but would need someone to with a such blob storage to test it out.

[shubhamdeodia]

@weyert Did you open sourced your POC ?

For one of the project I am working at my company, I need a remote cache server for Azure Blob Storage, I can provisioned blob storage to test the changes.

Would really appreciate your help on the same.

[djamaile]

@weyert when will you open-source it?

[weyert]

Please find it here: https://github.com/Tapico/tapico-turborepo-remote-cache :)

[weyert]

If this support will be added to turborepo you also need to think about companies that are for all kind of compliance reasons can not just directly connect to a cloud storage provider or might need to access via a VPN and/or IAP. For example, at work I am unable to directly access cloud storage and write to it from my computer system.

One of the reasons that I am worked on my above mentioned project to ease this problem. It's probably a good start for people that are concerned about not wanting to store build artefacts on Vercel. The project comes with binaries for amd64, windows, and arm and has docker images pre-build that can be used. I hope to find some free time to make an example how it can be ran via Cloud Run on Google Cloud.

If anyone want support for Azure Storage, please reach out to me and we can work on it together. I used a cloud abstraction library for Go that supports it already: https://github.com/graymeta/stow which supports the following:

• Microsoft Azure Blob Storage
• Openstack Swift (with auth v2)
• Oracle Storage Cloud Service
• SFTP

besides that ones already leveraged by the proxy server solution:

• Local (folders are containers, files are items)
• Amazon S3
• Google Cloud Storage

[fox1t]

You can find a Node.js/Fastify implementation here. It is an alpha, missing documentation and implementing only local storage. We plan to add support for various storage providers starting from S3. I am pretty excited about how easy it is to deploy and run your own remote caching server. :)

[fox1t]

We added s3 and the project is now stable and used in production :)

[trm217]

I think it would be cool if there was an actual documentation of the requirements for the Remote Caching Server API and not just a reference to the source-code.

[trm217]

@jaredpalmer I feel that this would provide great value for everyone who is not using Vercel.
I would like to help creating an actual documentation of the specification, however I am missing a bit of information.
For example I don't seem to find any info on the requirements for the login command to work with a remote cache.
Could you give more information on how all of this is meant to work together, without having to just read through most of the source-code?

[NullVoxPopuli]

I'd like this, too

[felixmosh]

Checkout github artifacts integration https://github.com/felixmosh/turborepo-gh-artifacts

[aperez900907]

It's any documentation on the cache server api part of turbo? I want to fully integrate with my local gitlab the monorepos build on turbo and the remote cache self-hosted. Making auth with the same bearer token from the gitlab user and translating groups as teams will be a match between the services.

[felixmosh]

I've made the same thing with GitHub, the proxy server runs inside the container and proxy to GitHub artifacts.
https://github.com/felixmosh/turborepo-gh-artifacts

[ThibautMarechal]

I build my own remote cache server (inpired by some of the existing ones listed above).
It's build with Remix. It store artifact either on local disk, azure or s3.
You can create users, teams, and also login with OpenID Connect.
I also handle the analytics events from turborepo so I can display statistics about the cache usage. (The same as Vercel show you in the dashboard)

Code source is here: https://github.com/ThibautMarechal/turborepo-remote-cache

[ThibautMarechal]

It doesn't support the artifact signature yet

[kevinsimper]

One approach that please.build uses it being able to define to commands like

RetrieveCommand="aws s3 cp s3://YOUR-OWN-CACHE-BUCKET/please/$CACHE_KEY -"

StoreCommand="aws s3 cp - s3://YOUR-OWN-CACHE-BUCKET/please/$CACHE_KEY"

https://please.build/cache.html

[NullVoxPopuli]

oh this would be super slick!

[sgammon]

Hey everybody,

We have decided to support TurboRepo via Buildless. Caches are all free right now during the beta and eventually we will have GitHub integration. If you sign up please tag me here so I can give you a free code for continued cache access beyond the beta and we could really use any feedback 🚀

[anthonyshew]

As noted by other users in this discussion, Turborepo supports customized caches as well as Vercel. 🥳

[NullVoxPopuli]

do any of these caches support turbo@v2?

[NullVoxPopuli]

hm -- maybe my issue is specifically with https://github.com/ducktors/turborepo-remote-cache then

[fox1t]

@NullVoxPopuli, can you please remind me what issue you are referring to? I would love to fix it!

[NullVoxPopuli]

yea, I'm getting a ton of

WARNING failed to contact remote cache: 
  Error making HTTP request: 
    HTTP status client error (400 Bad Request) for url 
      (https://<my-domain>/v8/artifacts/098ddf1ebd264906?slug=<my-project>)

in my logs

[NullVoxPopuli]

I also haven't been able to upgrade to the latest v1 for the same reason -- stuck on 1.12

[sgammon]

I believe we do at Buildless, yes.