Add key transform functions for massive rate limit improvements

README.md

@@ -24,14 +24,14 @@ As go plugins can be finicky to correctly compile and install, you may want to c
 > export GO111MODULE=on
 
 # Clone go-ipfs.
-> git clone https://github.com/ipfs/go-ipfs
-> cd go-ipfs
+> git clone https://github.com/ipfs/kubo
+> cd kubo
 
 # Pull in the datastore plugin (you can specify a version other than latest if you'd like).
-> go get github.com/ipfs/go-ds-s3/plugin@latest
+> go get github.com/3box/go-ds-s3/plugin@latest
 
 # Add the plugin to the preload list.
-> echo "s3ds github.com/ipfs/go-ds-s3/plugin 0" >> plugin/loader/preload_list
+> echo "s3ds github.com/3box/go-ds-s3/plugin 0" >> plugin/loader/preload_list
 
 # ( this first pass will fail ) Try to build go-ipfs with the plugin
 > make build
@@ -42,18 +42,18 @@ As go plugins can be finicky to correctly compile and install, you may want to c
 # Now rebuild go-ipfs with the plugin
 > make build
 
-# (Optionally) install go-ipfs
+# (Optionally) install go-ipfs aka Kubo
 > make install
 ```
 
 ## Detailed Installation
 
 For a brand new ipfs instance (no data stored yet):
 
-1. Copy s3plugin.so $IPFS_DIR/plugins/go-ds-s3.so (or run `make install` if you are installing locally).
+1. Copy `s3plugin.so` to `$IPFS_DIR/plugins/go-ds-s3.so` (or run `make install` if you are installing locally).
 2. Run `ipfs init`.
-3. Edit $IPFS_DIR/config to include s3 details (see Configuration below).
-4. Overwrite `$IPFS_DIR/datastore_spec` as specified below (*Don't do this on an instance with existing data - it will be lost*).
+3. Edit `$IPFS_DIR/config` to include s3 details for the **first** Datastore mount (see [Configuration](#configuration) below).
+4. Overwrite `$IPFS_DIR/datastore_spec` (*Don't do this on an instance with existing data - it will be lost*. See [Configuration](#configuration) below).
 
 ### Configuration
 
@@ -68,11 +68,12 @@ The config file should include the following:
         {
           "child": {
             "type": "s3ds",
-            "region": "us-east-1",
+            "region": "$bucketregion",
             "bucket": "$bucketname",
             "rootDirectory": "$bucketsubdirectory",
             "accessKey": "",
-            "secretKey": ""
+            "secretKey": "",
+            "keyTransform": "$keytransformmethod"
           },
           "mountpoint": "/blocks",
           "prefix": "s3.datastore",
@@ -81,6 +82,18 @@ The config file should include the following:
 ```
 
 If the access and secret key are blank they will be loaded from the usual ~/.aws/.
+
+The key transform allows you to specify how data is stored behind S3 keys. It must be one of the available methods:
+
+`default`
+- No sharding.
+
+`suffix`
+- Shards by storing block data at a key with a `data` suffix. E.g. `CIQJ7IHPGOFUJT5UMXIW6CUDSNH6AVKMEOXI3UM3VLYJRZUISUMGCXQ/data`
+
+`next-to-last/2`
+- Shards by storing block data based on the second to last 2 characters of its key. E.g. `CX/CIQJ7IHPGOFUJT5UMXIW6CUDSNH6AVKMEOXI3UM3VLYJRZUISUMGCXQ`
+
 If you are on another S3 compatible provider, e.g. Linode, then your config should be:
 
 ```json
@@ -93,12 +106,13 @@ If you are on another S3 compatible provider, e.g. Linode, then your config shou
         {
           "child": {
             "type": "s3ds",
-            "region": "us-east-1",
+            "region": "$bucketregion",
             "bucket": "$bucketname",
             "rootDirectory": "$bucketsubdirectory",
             "regionEndpoint": "us-east-1.linodeobjects.com",
             "accessKey": "",
-            "secretKey": ""
+            "secretKey": "",
+            "keyTransform": "$keytransformmethod"
           },
           "mountpoint": "/blocks",
           "prefix": "s3.datastore",
@@ -109,7 +123,7 @@ If you are on another S3 compatible provider, e.g. Linode, then your config shou
 If you are configuring a brand new ipfs instance without any data, you can overwrite the datastore_spec file with:
 
 ```
-{"mounts":[{"bucket":"$bucketname","mountpoint":"/blocks","region":"us-east-1","rootDirectory":"$bucketsubdirectory"},{"mountpoint":"/","path":"datastore","type":"levelds"}],"type":"mount"}
+{"mounts":[{"bucket":"$bucketname","mountpoint":"/blocks","region":"$bucketregion","rootDirectory":"$bucketsubdirectory"},{"mountpoint":"/","path":"datastore","type":"levelds"}],"type":"mount"}
 ```
 
 Otherwise, you need to do a datastore migration.
@@ -120,6 +134,12 @@ Feel free to join in. All welcome. Open an [issue](https://github.com/ipfs/go-ip
 
 This repository falls under the IPFS [Code of Conduct](https://github.com/ipfs/community/blob/master/code-of-conduct.md).
 
+### Local Development
+
+```
+go mod vendor
+```
+
 ### Want to hack on IPFS?
 
 [![](https://cdn.rawgit.com/jbenet/contribute-ipfs-gif/master/img/contribute.gif)](https://github.com/ipfs/community/blob/master/CONTRIBUTING.md)

go.mod

@@ -1,9 +1,9 @@
-module github.com/ipfs/go-ds-s3
+module github.com/3box/go-ds-s3
 
 require (
 	github.com/aws/aws-sdk-go v1.35.30
 	github.com/ipfs/go-datastore v0.5.1
-	github.com/ipfs/go-ipfs v0.11.0-rc1
+	github.com/ipfs/kubo v0.14.0
 )
 
 go 1.15