2. Setup for Local API Development

The setup steps are this page are meant to be performed sequentially.

Set up your keys

After cloning the API, you must set up your keys.py. Please pull the latest code if you did not just clone the repository. Within the directory into which you cloned the REopt API is a file called keys.py.template. First, make a copy of the keys.py.template file and rename it keys.py (in the same directory). Then, get a NREL Developer API key and replace the DEMO-KEY value within keys.py with your API key.

Solvers

v3 of the REopt API may be used with the following open source solvers: HiGHS, Cbc, and SCIP. To use one of these, you must specify which solver in the Settings.solver_name input; specify the name exactly as listed above, as a string input. If you have an Xpress license, you may also use that with the appropriate installation instructions and license server setup in separate Dockerfile.xpress and docker-compose.xpress.yml files. It is advised that you reach out to NREL to help setting up Xpress or another commercial solver.

Install and build docker

Once you have your keys.py and solver set up, install docker (https://www.docker.com/get-started) and create an account so that you are able to pull images from docker hub.

Build docker images for v3 and higher

Before building docker images, make sure the docker application is open. Choose the right instructions for your purpose:

• Using the API (v3+) with the officially registered REopt Julia package
• Using the API (v3+) with a REopt.jl feature branch or locally-hosted repository
• Using the v1-v2 REopt API

If using the API with the officially registered REopt Julia package

The officially-registered REopt Julia package is the master branch here and the package imported by defaulted when a user does add REopt in Julia. See the next section if you are instead wanting to access a locally "dev'ed" branch of REopt.jl or a feature branch of REopt.jl on GitHub.

Build docker images, by entering the following in a terminal in the top directory of the REopt_API:

docker compose build

or e.g. for a separate Xpress Docker setup:

docker compose -f docker-compose.xpress.yml build

This step is only required the first time or if the images have changes or were removed. This will take some time to build (~10 minutes), and when it finishes you should see the images listed when you type docker images into a terminal or view the images in the Docker Desktop app.

Start the containers (anytime the containers are stopped or removed, or your computer sleeps/hibernates/restarts) with:

docker compose up

or e.g. for a separate Xpress Docker setup:

docker compose -f docker-compose.xpress.yml up

This should be pretty quick, but you have to wait for the containers to start running (Check Docker Desktop, another Docker UI, or type docker ps into another terminal to see a list of five running containers).

Once built, all containers should appear as green and "running" in the docker GUI.

If using the API with a REopt.jl feature branch or locally-hosted repository

Spin up docker with a feature branch of the REopt Julia

1. Make your changes to the REopt Julia package as needed and push your feature branch to GitHub (should appear here; you will need to have write access to do this.)
2. Point the julia_src environment of the REopt API to the feature branch of the Julia package by running the following from the julia_src directory:

julia --project=.
using Pkg
pkg> add REopt#branchname

(replace branchname with feature branch name from https://github.com/NREL/REopt.jl/branches)

The step above will modify the REopt_API julia_src/Manifest.toml file. Any local API calls or unit tests you run will now use specified feature branch of REopt.jl.

3. Spin up docker containers by changing back to the top-level directory of REopt_API then do (NREL users may need to suspend VPN):

docker compose build

4. Start the containers (anytime the containers are stopped or removed, or your computer sleeps/hibernates/restarts) with:

docker compose up

Once built, all containers should appear as green and "running" in the docker GUI.

NOTE: Step 2 will modify the julia_src/Manifest.toml file for the REoptLite entry. DO NOT COMMIT THESE CHANGES (with the exception of instances in which you wish to use GitHub's unit tests with your changes, in which case you should eventually undo them as shown below).

5. Once you are done testing your local changes to the Julia package, reset the julia_src environment with: pkg> add REopt (from the julia_src directory with that environment activated)

Spin up docker with a local copy of the Julia package

If you do not have write access to REopt.jl or want to use a locally-modified version of REopt.jl, follow these instructions for spinning up docker.

Instead of using the default docker-compose.yml file when developing the Julia package locally in concert with the API, one must use the docker-compose.nojulia.yml file for starting the the API components (Redis, Postgresql, Django, Celery, and a solver license server if you have one). As the name implies, the docker-compose.nojulia.yml does not start a Julia service. Instead, to test the API with local changes to the Julia package one must start the Julia HTTP service locally in an environment that points towards your local copy of the Julia package.

1. Make your changes to the Julia package and/or the API code as needed.
2. docker-compose -f docker-compose.nojulia.yml up (This step can really be run at any time, unless you are modifying Celery tasks such as run_jump_model, which requires restarting the Django and Celery services to pick up on changes.)
3. Point the julia_src environment to your local copy of the Julia package by running the following from the julia_src directory:

julia --project=.
using Pkg
Pkg.develop("REopt")

4. Exit the Julia session from the previous step and run julia --project=. http.jl (or, in another julia terminal include(http.jl). This will start the Julia HTTP server that will provide the reopt endpoint, which will use your local copy of the REopt Julia package to run_reopt.

NOTE: Step 3 will modify the julia_src/Manifest.toml file for the REoptLite entry. DO NOT COMMIT THESE CHANGES (with the exception of instances in which you wish to use GitHub's unit tests with your changes, in which case you should eventually undo them as shown below).

Once you are done testing your local changes to the Julia package, reset the julia_src environment with:

julia --project=julia_src
using Pkg
Pkg.free("REopt")

Stop/Remove the docker containers

Docker containers are meant to be ephemeral, only persisting for the required development and testing. When you are finished with them, it is good practice to stop and remove them.

Stop your docker containers with (cntrl/cmd+c).

The command below removes containers, and removes the database which is stored in a docker volume (-v). For open source solver users:

docker compose down -v

For Xpress users:

docker compose -f docker-compose.xpress.yml down -v

To remove all unused containers, networks, images (both dangling and unreferenced), and optionally, volumes, run (see documentation:

docker system prune

The next time you want to run the app, start at the build step if the images have changes or were removed. If not, start at the up step.

Docker setup FAQ and troubleshooting

Out-of-memory issues

If you installed docker-compose via Docker Desktop the default memory limits for the celery container maybe too low (2GB for Mac for example). The optimization process can take more than 3 GB of memory, which will lead to a container killing the process. Therefore, we recommend increasing the Docker memory limit to at least 4 GB. You can do this in the docker GUI via Resources > Advanced.

Address already in use

If you are already running PostgreSQL or Redis when you start the docker containers then you may get a conflict with the ports that PostgreSQL and Redis use. To fix this you can either shut down your PostgreSQL and/or Redis servers or change the port numbers in the docker-compose.yml file. For more on ports see the Docker documentation.

If you are having trouble getting the API setup with Docker after you've attempted it the first time, it is suggested to start from scratch with the images and containers:

Other useful docker commands

To check if all the containers are running as expected, run the following command:

docker ps

You can interactively run commands in a docker container with:

docker exec -it <container-name> /bin/bash

where container names can be found via docker ps. For more see the Docker documentation.

For instructions on running tests when logged into the celery container see Develop and Test the API (v3). TODO: Update

If you are having trouble getting the API setup with Docker

After you've attempted it the first time, it is suggested to start from scratch with the images and containers:

Stop and remove all docker containers and volumes

For open source solver:

docker compose down -v

For Xpress solver:

docker compose -f docker-compose.xpress.yml down -v

Delete all docker images and build cache (WARNING: this will remove any images even if not associated with the REopt API)

docker system prune -a

Remove selected images to avoid wiping non-REopt API images

First get a list of the images

docker images

Then do this for each of the REopt API images:

docker image rm [image_name]

Check that everything is wiped

docker system df

You should see zeros across the board (unless you only removed selected images). There may be a dangling volume still, in which case you can:

docker volume prune

Windows docker image build issue Specifically for Windows operating machines, if you encounter an error which includes “bash^M”, run the following line of code in git bash terminal inside of your repository directory. You may need to repeat for each branch you work on... 

sed -i -e 's/\r$//' bin/wait-for-it.bash