feat: add more command line inputs

[danielolsen]

Purpose

• Adds command-line specification of the number of linearization segments to use (Closes Add specification of number of linearization segments to command-line call.py #98).
• Adds command-line specification of the julia environment to use (Closes Allow user to specify the Julia environment to use via call.py #104)
• Actually use the solver keyword arguments (added in refactor: remove Gurobi requirement #106 as a parameter but not actually used at all)
• Properly interpret solvers that are specified from the command-line, regardless of upper or lower case (cleanup from Define glpk launcher and expose command line and api option #112)
• Refactor how Launcher objects are used to minimize redundant code.

What is the code doing

• In parser.py: adding two new command-line arguments.
• In call.py: passing command-line arguments (including new ones) directly to the Launcher init.
• In launchers.py:
  • All parameters passed for launching a simulation go to the Launcher init, to minimize duplication in the child launch_scenario methods.
  • A new init_julia method is added which will start a Julia session in the specified environment and import a given list of Julia packages. To be able to start in a given environment, we need to change the way we start the Julia sessions from from julia.api import Julia; Julia(compiled_modules=False) to from julia.api import LibJulia; api = LibJulia.load(); api.init_julia(["--compiled-modules=no"]); these are equivalent, see https://pyjulia.readthedocs.io/en/latest/troubleshooting.html#turn-off-compilation-cache. This lets us add another command-line argument to start in a given project folder (environment), see https://julialang.github.io/Pkg.jl/v1.5/environments/#Using-someone-else's-project
  • A new parse_and_return_runtime parse_runtime method is added to minimize duplication in the child launch_scenario methods.
  • The launcher lookup is fixed so that "GLPK" works, in addition to "glpk"

Usage Example

First, we'll set up a new Julia environment in an arbitrary folder, and install REISE and GLPK into this environment:

julia> import Pkg
julia> Pkg.activate(raw"C:\Users\DanielOlsen\repos\bes\scenario_3216\REISE_GLPK")
julia> Pkg.develop(path=raw"C:\Users\DanielOlsen\repos\bes\REISE.jl")
[...install truncated...]
julia> Pkg.add("GLPK")
[...install truncated...]
julia> exit()

Then, let's use all of our new command line parameters, executing them inside this new environment:

PS C:\Users\DanielOlsen\repos\bes\REISE.jl> python pyreisejl\utility\call.py -s '2016-01-01 00:00:00' -e '2016-01-01 05:00:00' -int 2 -i C:\Users\DanielOlsen\repos\bes\scenario_3216 -j C:\Users\DanielOlsen\repos\bes\scenario_3216\REISE_GLPK -l 3 --solver GLPK
Validation complete!
Launching scenario with parameters:
{'num_segments': 3, 'interval': 2, 'n_interval': 3, 'start_index': 1, 'input_dir': 'C:\\Users\\DanielOlsen\\repos\\bes\\scenario_3216', 'execute_dir': None, 'threads': None, 'julia_env': 'C:\\Users\\DanielOlsen\\repos\\bes\\scenario_3216\\REISE_GLPK', 'solver_kwargs': None}
INFO: threads not supported by GLPK, ignoring
Reading from folder: C:\Users\DanielOlsen\repos\bes\scenario_3216
...loading case.mat
...loading demand.csv
...loading hydro.csv
...loading wind.csv
...loading solar.csv
File case_storage.mat not found in C:\Users\DanielOlsen\repos\bes\scenario_3216
All scenario files loaded!
linearizing
All preparation complete!
Redirecting outputs, see stdout.log & stderr.err in outputfolder
Run time: 0:01:37

This works even if GLPK is not installed in your default environment, which you can check with a simple import in fresh julia session:

julia> import GLPK
ERROR: ArgumentError: Package GLPK not found in current path:
- Run `import Pkg; Pkg.add("GLPK")` to install the GLPK package.

Time to review

30 minutes.

[jenhagg]

I got a warning during test run, just fyi. Each line is from stderr in the api response, and I removed a duplicate from line 74. The scenario still ran successfully though.

/app/pyreisejl/utility/helpers.py:72: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.",
  'data = np.array([f[r][()] for r in data.flat])',

[jenhagg]

I might call this just parse_runtime since the return is kinda implied

[danielolsen]

Done

[jenhagg]

Since we don't use the gurobi import directly, you could do _, REISE = ...

[danielolsen]

Done.

[danielolsen]

I got a warning during test run, just fyi. Each line is from stderr in the api response, and I removed a duplicate from line 74. The scenario still ran successfully though.

/app/pyreisejl/utility/helpers.py:72: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.",
  'data = np.array([f[r][()] for r in data.flat])',

Yeah, this is from our custom parse-hdf5-matfile-to-nested-dict-of-arrays script, and it's not new. Everything still works fine, but we'll probably want to update the call to avoid this warning in the future.

[jenhagg]

LGTM. I was able to run simulations with both glpk and gurobi successfully, and your test of the new arguments looks sufficient.

[jenhagg]

One more question: is the default of 1 for num_segments consistent with the current behavior?

[danielolsen]

One more question: is the default of 1 for num_segments consistent with the current behavior?

Yes, see

REISE.jl/src/REISE.jl

Line 48 in 332e3ca

num_segments::Int=1, interval::Int, n_interval::Int, start_index::Int,

[BainanXia]

I set up my local environment for running REISE.jl on my mac and run similar tests as @danielolsen posted. It works!

bxia REISE.jl (daniel/more_command_line_inputs) $ python pyreisejl/utility/call.py -s '2016-01-01 00:00:00' -e '2016-01-01 05:00:00' -int 2 -i "/Users/bainanxia/OneDrive - Gates Ventures/Documents/local_scenario_runs/scenario_3217" -l 3 --solver GLPK
Validation complete!
Launching scenario with parameters:
{'num_segments': 3, 'interval': 2, 'n_interval': 3, 'start_index': 1, 'input_dir': '/Users/bainanxia/OneDrive - Gates Ventures/Documents/local_scenario_runs/scenario_3217', 'execute_dir': None, 'threads': None, 'julia_env': None, 'solver_kwargs': None}
INFO: threads not supported by GLPK, ignoring
Reading from folder: /Users/bainanxia/OneDrive - Gates Ventures/Documents/local_scenario_runs/scenario_3217
...loading case.mat
...loading demand.csv
...loading hydro.csv
...loading wind.csv
...loading solar.csv
File case_storage.mat not found in /Users/bainanxia/OneDrive - Gates Ventures/Documents/local_scenario_runs/scenario_3217
All scenario files loaded!
linearizing
All preparation complete!
Redirecting outputs, see stdout.log & stderr.err in outputfolder
Run time: 0:01:28