Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix add files error #14

Draft
wants to merge 12 commits into
base: main
Choose a base branch
from
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
bin
cids
file
26 changes: 26 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,18 @@

Script to manage the IPFS files. It can be used to sync the CID's between two nodes

## Build

```
go build -o bin/ipfs-mgm cmd/cli/ipfs-mgm.go
```

or use the Dockerfile

```
docker build -t ipfs-mgm .
```

## Install

### Manually
Expand Down Expand Up @@ -35,6 +47,14 @@ ipfs-mgm sync --help
ipfs-mgm sync -s <SOURCE URL> -d <DESTINATION URL>
```

#### Transfer all files using batches and cooldown

```bash
ipfs-mgm sync -s <SOURCE URL> -d <DESTINATION URL> -f cids -c 5 -b 100
```
*INFO: `-b 100` will transfer the files in batches of 100 in paralel and wait 5 seconds (`-c 5`) in between the batches to avoid overloading the IPFS nodes*


#### Transfer only specific files from one IPFS node to another:

```bash
Expand All @@ -60,6 +80,12 @@ The easiest way is to use the built docker image
docker run -it ghcr.io/graphprotocol/ipfs-mgm sync --help
```

When using a local file

```bash
docker run -it -v $PWD:/app ghcr.io/graphprotocol/ipfs-mgm sync -s <SOURCE_ENDPOINT> -d <SOURCE_ENDPOINT> -f /app/file
```

## TODO:

- [ ] Implement async calls by creating a worker queue in batches
Expand Down
46 changes: 5 additions & 41 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -3,47 +3,11 @@ module github.com/graphprotocol/ipfs-mgm
go 1.18

require (
github.com/benbjohnson/clock v1.3.0 // indirect
github.com/blang/semver/v4 v4.0.0 // indirect
github.com/crackcomm/go-gitignore v0.0.0-20170627025303-887ab5e44cc3 // indirect
github.com/decred/dcrd/dcrec/secp256k1/v4 v4.1.0 // indirect
github.com/fsnotify/fsnotify v1.6.0 // indirect
github.com/hashicorp/hcl v1.0.0 // indirect
github.com/spf13/cobra v1.7.0
golang.org/x/time v0.1.0
)

require (
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/ipfs/boxo v0.12.0 // indirect
github.com/ipfs/go-cid v0.4.1 // indirect
github.com/ipfs/go-ipfs-api v0.7.0 // indirect
github.com/klauspost/cpuid/v2 v2.2.3 // indirect
github.com/libp2p/go-buffer-pool v0.1.0 // indirect
github.com/libp2p/go-flow-metrics v0.1.0 // indirect
github.com/libp2p/go-libp2p v0.27.8 // indirect
github.com/magiconair/properties v1.8.7 // indirect
github.com/minio/sha256-simd v1.0.0 // indirect
github.com/mitchellh/go-homedir v1.1.0 // indirect
github.com/mitchellh/mapstructure v1.5.0 // indirect
github.com/mr-tron/base58 v1.2.0 // indirect
github.com/multiformats/go-base32 v0.1.0 // indirect
github.com/multiformats/go-base36 v0.2.0 // indirect
github.com/multiformats/go-multiaddr v0.8.0 // indirect
github.com/multiformats/go-multibase v0.2.0 // indirect
github.com/multiformats/go-multicodec v0.9.0 // indirect
github.com/multiformats/go-multihash v0.2.3 // indirect
github.com/multiformats/go-multistream v0.4.1 // indirect
github.com/multiformats/go-varint v0.0.7 // indirect
github.com/pelletier/go-toml/v2 v2.1.0 // indirect
github.com/spaolacci/murmur3 v1.1.0 // indirect
github.com/spf13/afero v1.9.5 // indirect
github.com/spf13/cast v1.5.1 // indirect
github.com/spf13/cobra v1.7.0 // indirect
github.com/spf13/jwalterweatherman v1.1.0 // indirect
github.com/spf13/pflag v1.0.5 // indirect
github.com/spf13/viper v1.16.0 // indirect
github.com/subosito/gotenv v1.6.0 // indirect
golang.org/x/crypto v0.9.0 // indirect
golang.org/x/sys v0.12.0 // indirect
golang.org/x/text v0.13.0 // indirect
google.golang.org/protobuf v1.30.0 // indirect
gopkg.in/ini.v1 v1.67.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
lukechampine.com/blake3 v1.1.7 // indirect
)
528 changes: 2 additions & 526 deletions go.sum

Large diffs are not rendered by default.

Loading
Loading