Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow the user to define input and output file names #639

Open
RuaridhMacd opened this issue Feb 27, 2024 · 3 comments
Open

Allow the user to define input and output file names #639

RuaridhMacd opened this issue Feb 27, 2024 · 3 comments
Assignees
Labels
enhancement New feature or request

Comments

@RuaridhMacd
Copy link
Collaborator

It would be useful to be able to define different file names for input and output files. This will allow for some more flexibility when running large sequences of cases.

I propose adding an optional argument to all input loading and output writing functions with the current filenames as the default values. We'll also write a short log of where the outputs were saved to, to help automated parsing of the outputs later.

@RuaridhMacd RuaridhMacd self-assigned this Feb 27, 2024
@JesseJenkins JesseJenkins added the enhancement New feature or request label Mar 25, 2024
@gschivley
Copy link
Contributor

@lbonaldo and I have talked about some related improvements like allowing .gz or parquet input files to reduce file size, or letting users build a database with tables that can be reused across cases. Some input files can get large and/or might be duplicated across cases (and even periods).

I recently wrote some python functions that use DuckDB to load data from either a data file (.csv or .parquet) or a database table (.sqlite or .duckdb). The top-level function (load_data) takes arguments of the data location (folder or database path), the file or table name, and an optional sql query. Something similar could be implemented in GenX, letting users define input file names/locations and use different file types.

@mmutic
Copy link
Collaborator

mmutic commented Jul 1, 2024

Rebased branch to develop

@mmutic
Copy link
Collaborator

mmutic commented Jul 12, 2024

Update 7/12 -- Added default input names and file paths so that users can specify names of their input files in a YAML file called "input_settings.yml" in the settings folder of their case. I added an example (10_three_zones_define_input) with this file in it. I tested this for run_genx_simple and will test on multistage soon. I'm working on using DuckDB so that users can input files of types other than CSV (currently input files still need to be CSV).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

When branches are created from issues, their pull requests are automatically linked.

5 participants