Skip to content

Latest commit

 

History

History
26 lines (16 loc) · 1.34 KB

README.md

File metadata and controls

26 lines (16 loc) · 1.34 KB

Oregon Water Data Portal ETL

The Oregon Water Data portal crawls or ingests data from multiple different Oregon sources and exposes them via a Sensorthings and OGC API Features endpoint.

How to use

  1. Copy the .env.example file to .env and change the URL to your desired domain.
  2. Spin up the infrastructure with docker
docker compose --env-file .env --profile production up
  1. Once you start the database and the frost server, you will likely want to add extra indices to make it faster. To do this run the init script

Access the frontend at localhost:8999 and the API at localhost:8999/oapi, or edit .env and deploy at your own URL.

  1. Start Caddy with make caddy if you want to get https in production.

Development

When developing you don't need production services so you can just run docker compose up (without --profile production) and then run dagster with dagster dev

  • You must use python 3.12 or below since 3.13 is not supported by Dagster currently
  • This project uses uv since it makes it easier to manage the python version
  • It is sometimes helpful to use screen to start a background shell, run dagster dev then detach the screen. You can then reattach to it at a later time with screen -r so it runs in the background but is still accessible via the cli.