Skip to content

Setting up your Estuary Node

Jump to bottom
philbed edited this page Sep 19, 2022 · 1 revision

Setup

This section will guide you through the process of building Estuary. Read all instructions carefully and check the troubleshooting section if you run into any issues.

Requirements and Dependencies

  1. Go (Version 1.15 or higher)
  2. jq
  3. hwloc
  4. OpenCL
  5. Git

Building

  1. Make sure you have all of the requirements and dependencies (listed above) installed and clone this repo. git clone https://github.com/application-research/estuary

  2. Now move into the estuary directoy.

  3. In the estuary directory, run make clean all. You should get an output something like this:

    ...
go build  -ldflags="-X=main.appVersion=v0.1.8-2-g72ed2ba" -o bsget ./cmd/bsget
go: downloading github.com/cheggaaa/pb/v3 v3.0.8
go: downloading github.com/VividCortex/ewma v1.1.1

If running make clean all returns any errors, it is likely a problem with dependencies. Check the troubleshooting section to see if your specific error is documented, otherwise check that your dependencies are installed correctly.

Contribute

Estuary welcomes contributions to our open source projects on GitHub.

This section will help you get started with running a local Estuary node in a development environment.

Before submitting patches or additions, please refer to each projects style and contribution guidelines. In general, we follow the "fork-and-pull" Git workflow.

  1. Fork the repo on GitHub.
  2. Clone the project onto your own machine. git clone https://github.com/application-research/estuary
  3. Commit changes to your own branch.
  4. Push your work back up to your fork.
  5. Submit a pull request.

Note: Be sure to merge the latest from "upstream" before making a pull request.

Issues

Feel free to submit issues and enhancement requests.

Please use Estuary support to report specific bugs and errors.

Development

  1. First, follow the steps in the building section to get Estuary cloned and built.

  2. Initialize the database and generate your access token.

    ./estuary setup --username=<uname> --password=<pword>

    Be sure to save the auth token this command returns, it will be used to do almost all of the API calls.

  3. Set the FULLNODE_API_INFO environment variable to a synced Lotus node. This can either be a local Lotus node or a public endpoint, wss://api.chain.love for example.

    $ export FULLNODE_API_INFO=wss://api.chain.love

  4. Start your Estuary node.

    $ ./estuary --logging

    This will output:

    Wallet address is:  <your_estuary_address_printed_here>
2021-09-16T13:32:48.463-0700    INFO    dt-impl impl/impl.go:145        start data-transfer module
/ip4/192.168.1.235/tcp/6744/p2p/12D3KooWEb5wbWf3KcExdLUMp4x9NqzyaMcBaYbxZ9V6RUNwvpX8
/ip4/127.0.0.1/tcp/6744/p2p/12D3KooWEb5wbWf3KcExdLUMp4x9NqzyaMcBaYbxZ9V6RUNwvpX8
2021-09-16T13:32:48.464-0700    INFO    estuary estuary/replication.go:687      queueing all content for checking: 0

   ____    __
  / __/___/ /  ___
 / _// __/ _ \/ _ \
/___/\__/_//_/\___/ v4.5.0
High performance, minimalist Go web framework
https://echo.labstack.com
____________________________________O/_______
                                    O\
⇨ http server started on [::]:3004

    If you get an error message like too many open files, increase the number of open files allowed using ulimit -n 10000.

Initialize and Start a Shuttle

Estuary stores data on IPFS before replicating it to the Filecoin Network. When you store data using Estuary, that data will first go to an estuary-shuttle node that utilizes IPFS as hot-storage before replication to the Filecoin Network begins.

  1. Build the shuttle binary.

    $ make estuary-shuttle

  2. Initialize a shuttle node using the Estuary API endpoint /admin/shuttle/init or the admin UI at /admin/shuttle.

    $ curl -H "Authorization: Bearer ESTb43c2f9c-9832-498a-8300-35d9c4b8c16eARY" -X POST localhost:3004/admin/shuttle/init
{"handle":"SHUTTLE4e8b1770-326c-4c95-9976-7cc1ee12244bHANDLE","token":"SECRET7528ab25-1266-4fa4-86cf-719a43bbcb4fSECRET"}

  3. Using the output from the above command, start a shuttle node in development mode.

    $ ./estuary-shuttle --dev --estuary-api=localhost:3004 --auth-token=SECRET7528ab25-1266-4fa4-86cf-719a43bbcb4fSECRET --handle=SHUTTLE4e8b1770-326c-4c95-9976-7cc1ee12244bHANDLE --logging --host=localhost:3005

    This will output:

    Wallet address is:  <your_estuary-shuttle_address_printed_here>
2021-09-16T14:47:54.353-0700    INFO    dt-impl impl/impl.go:145        start data-transfer module
2021-09-16T14:47:54.416-0700    INFO    shuttle estuary-shuttle/main.go:1060    refreshing 0 pins

   ____    __
  / __/___/ /  ___
 / _// __/ _ \/ _ \
/___/\__/_//_/\___/ v4.5.0
High performance, minimalist Go web framework
https://echo.labstack.com
____________________________________O/_______
                                    O\
⇨ http server started on [::]:3005
2021-09-16T14:47:54.416-0700    INFO    shuttle estuary-shuttle/rpc.go:54       sending rpc message: ShuttleUpdate
2021-09-16T14:47:54.417-0700    INFO    shuttle estuary-shuttle/main.go:466     connecting to primary estuary node
2021-09-16T14:47:54.417-0700    INFO    shuttle estuary-shuttle/main.go:519     sending hello   {"hostname": "", "address": "<your_estuary-shuttle_address_printed_here>", "pid": "12D3KooWHag4gY8fQkjQ8Rgs5v6Fb4TpP3LXm8xvhVX3GsVKNwUW"}

Note: The above commands can be repeated to create and run more shuttle nodes.

API

At this point you may begin adding files to Estuary via the API described in the documentation, or using the golang client creek.

Troubleshooting

Make sure all dependencies listed above are installed correctly. Here are some of the issues that you may encounter while building Estuary.

route ip+net: netlinkrib: too many open files

Error: /ERROR basichost basic/basic_host.go:328 failed to resolve local interface addresses {"error": "route ip+net: netlinkrib: too many open files"}

This error means you do not have enough open file handles available.

Solution

Update this with the following command: ulimit -n 10000

Missing hwloc on M1 Macs

lhwloc is used by libp2p-core. Estuary uses libp2p for the majority of its features including network communication, pinning, replication and resource manager.

Error: ld: library not found for -lhwloc

Solution

For M1 Macs, follow these steps.

  1. Unistall Rust if you have it, it will clash with rustup in the next step.
  2. Run brew install go bzr jq pkg-config rustup hwloc.
  3. Run export LIBRARY_PATH=/opt/homebrew/lib
  4. Now continue on with the building process using the steps above.

Cannot find -lfilcrypto collect2

Related issue here.

Error: # github.com/filecoin-project/filecoin-ffi/generated /usr/bin/ld: skipping incompatible extern/filecoin-ffi/generated/../libfilcrypto.a when searching for -lfilcrypto /usr/bin/ld: skipping incompatible extern/filecoin-ffi/generated/../libfilcrypto.a when searching for -lfilcrypto /usr/bin/ld: skipping incompatible extern/filecoin-ffi/generated/../libfilcrypto.a when searching for -lfilcrypto /usr/bin/ld: skipping incompatible extern/filecoin-ffi/generated/../libfilcrypto.a when searching for -lfilcrypto /usr/bin/ld: skipping incompatible extern/filecoin-ffi/generated/../libfilcrypto.a when searching for -lfilcrypto /usr/bin/ld: cannot find -lfilcrypto collect2: error: ld returned 1 exit status make: *** [Makefile:67: estuary] Error 2

Solution

Run RUSTFLAGS="-C target-cpu=native -g" FFI_BUILD_FROM_SOURCE=1 make clean deps bench

Solution to a related issue is here.

API Documentation | API Documentation with Swagger | Estuary Database Schema

Clone this wiki locally