Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: update local dev guide #12585

Merged
merged 6 commits into from
Feb 14, 2025
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 50 additions & 12 deletions docs/developers.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,38 +77,76 @@ We suggest partially compiling DataHub according to your needs:
./gradlew :docs-website:serve
```

## Deploying Local Versions
## Deploying Local Versions
This guide explains how to set up and deploy DataHub locally for development purposes.

Run just once to have the local `datahub` cli tool installed in your $PATH
### Initial Setup
Before you begin, you'll need to install the local `datahub` CLI tool:

```shell
cd smoke-test/
cd metadata-ingestion/
python3 -m venv venv
source venv/bin/activate
pip install --upgrade pip wheel setuptools
pip install -r requirements.txt
cd ../
```

Once you have compiled & packaged the project or appropriate module you can deploy the entire system via docker-compose by running:
### Deploying the Full Stack

Deploy the entire system using docker-compose:
```shell
./gradlew quickstart
./gradlew quickstartDebug
```

Replace whatever container you want in the existing deployment.
I.e, replacing datahub's backend (GMS):
Access the DataHub UI at `http://localhost:9002`

### Refreshing the Frontend

To run and update the frontend with local changes, open a new terminal and run:
```shell
(cd docker && COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker compose -p datahub -f docker-compose-without-neo4j.yml -f docker-compose-without-neo4j.override.yml -f docker-compose.dev.yml up -d --no-deps --force-recreate --build datahub-gms)
cd datahub-web-react
yarn install && yarn start
```
The frontend will be available at `http://localhost:3000` and will automatically update as you make changes to the code.

Running the local version of the frontend
### Refreshing GMS

To refresh the GMS (Generalized Metadata Service) with local changes:
```shell
(cd docker && COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker compose -p datahub -f docker-compose-without-neo4j.yml -f docker-compose-without-neo4j.override.yml -f docker-compose.dev.yml up -d --no-deps --force-recreate --build datahub-frontend-react)
./gradlew :metadata-service:war:build -x test --parallel && docker restart datahub-datahub-gms-debug-1
```

### Refreshing the CLI

If you haven't set up the CLI for local development yet, run:

```commandline
./gradlew :metadata-ingestion:installDev
cd metadata-ingestion
source venv/bin/activate
```

Once you're in `venv`, your local changes will be reflected automatically.
For example, you can run `datahub ingest -c <file>` to test local changes in ingestion connectors.

To verify that you're using the local version, run:

```commandline
datahub --version
```

Expected Output:
```commandline
acryl-datahub, version unavailable (installed in develop mode)
```

### Refreshing Other Components

To refresh other components with local changes, just run:
```commandline
./gradlew quickstartDebug
```


## IDE Support

The recommended IDE for DataHub development is [IntelliJ IDEA](https://www.jetbrains.com/idea/).
Expand Down