Support synchronizing with a custom script / executable

[dflems]

Until support exists for more storage systems, the only options for using Rome are S3 and the local cache. For security and other reasons, we can't easily use S3 with our current setup. I propose that Rome adds support for synchronizing using a custom script or executable with a well-defined interface. A status code of 0 would indicate success and standardized error codes could indicate other statuses (file not found, etc). This way, people can plug in the script to add support for whatever storage system they want no matter how bespoke or esoteric it may be:

Supply custom-exec option in Romefile

[Cache]
local = ~/Library/Caches/Rome
custom-exec = ./my-custom-sync-script.sh

rome executes the script to handle actions

# upload to remote
./my-custom-sync-script.sh upload <local-path> <remote-path>

# download from remote
./my-custom-sync-script.sh download <remote-path> <local-path>

# probe remote for existence of file
./my-custom-sync-script.sh probe <remote-path>

Trivial example of a custom HTTP sync script

#!/bin/bash
set -e

ACTION="$1"
BASE_URL="http://my.server.internal/rome_cache"

if [ "$ACTION" == "upload" ]; then
  # make PUT request to save file on remote
  LOCAL_PATH="$2"
  REMOTE_PATH="$3"
  curl -sf -X PUT "$BASE_URL$REMOTE_PATH" -d "@$LOCAL_PATH"

elif [ "$ACTION" == "download" ]; then
  # make GET request for file, save to specified output path
  REMOTE_PATH="$2"
  OUTPUT_PATH="$3"
  curl -sf "$BASE_URL$REMOTE_PATH" -o "$OUTPUT_PATH"

elif [ "$ACTION" == "probe" ]; then
  # make HEAD request to check for existence of file
  REMOTE_PATH="$2"
  curl -sfI "$BASE_URL$REMOTE_PATH" > /dev/null
else
  # unsupported command
  exit 1
fi

[tmspzz]

Hello,

thanks for opening an issue.

Until support exists for more storage systems, the only options for using Rome are S3 and the local cache.

You can use Minio or Ceph, or any other S3 compatible store. You don't have to forcibly use S3.

Nevertheless this is a good idea.

Rome supports a variety of switches, including:

• --platform (restrict operations to the specified platforms)
• --cache-prefix (a prefix to append to the remote paths)
• -v (verbose)

I have the feeling these should be passed on, just in case.

I would be happier if the communication between rome and the "engines"
was based on JSON for I/O rather than relying on exit codes.

[tmspzz]

For example the scripts can be called with:

• the current executing command: one of upload , download or list
• all the parameters rome was called with

as parameters to the script:

eg. ./my-custom-sync-script.sh download --cache-prefix somePrefix -v --platforms ios,mac

the script would then:

• read from stdin the JSON representing the data
• write to stdout write output data

This would mean that you might not be able to write random stuff to stdout, or the random lines should be prefixed with some escape sequence like # line of text

[dflems]

Forwarding the original args makes sense, as does stdout/stderr.

You can use Minio or Ceph, or any other S3 compatible store. You don't have to forcibly use S3.

We have a filestore service (sadly not S3 compatible) that is by default read-only on our VPN or office network, and CI machines are credentialed to write to it. That seemed like the ideal producer/consumer flow, but we need something like this script-runner support to handle it.

Another solution could be a transparent pass-through proxy to our internal storage service that emulates the S3 API. That was my original idea but this one seemed a bit cleaner and extensible for other storage solutions, not just my own :)

[dflems]

Another option that could suit our needs (and maybe others) is simple HTTP GET/PUT/HEAD with optional credentials.

In this case, you would provide a base URI, for example https://my.server/base. If you wanted to upload the file foo/bar/baz.zip, the URI you would interact with would then be https://my.server/base/foo/bar/baz.zip.

• list command would make HEAD request
• download command would make a GET request
• upload command would make a PUT request

You can assume normal HTTP status codes indicate the success or failure of the request.

You can optionally supply a credentials file (json or yaml) via configuration or environment variable that indicates that additional headers that the request should be decorated with:

Authorization: my-special-token

In our case, we could use that for uploading content on the CI side.

[tmspzz]

Great. Now two questions:

• Who is going to develop the feature? Can you help? I am busy with Rome incorrectly assumes supported platform #135 atm
• What company is this for? Is this for your own project?

I'm asking this in case, you know, it's down to me to work for free (yes, of course I care for the adoption of the project but you know, I have a day job too and just so many hours at night)

See my roadmap: https://github.com/blender/Rome/projects/1

Another option that could suit our needs (and maybe others) is simple HTTP GET/PUT/HEAD with optional credentials.

How about .netrc ?

[tmspzz]

Actually if all you want is to add an Authorization header then rome should not even care about your credentials. In shorts you could just specify in some file:

headers:
 - Authorization Basic: SGVsbG86dGhlcmUK

and rome will just add this header

[dflems]

@blender: I don't have a lot of time at the moment, but if I can find some free time I can look into doing it. I'd need to learn haskell first though :)

[tmspzz]

Leaning Haskell is rewarding in itself ;)

[tmspzz]

@dflems Thanks to @BalestraPatrick you can now use whatever backend you want. #185

You can try the pre-release at: https://github.com/blender/Rome/releases/tag/v0.22.0.59