Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Toolkit #30

Open
5 tasks
zachsa opened this issue Mar 10, 2023 · 0 comments
Open
5 tasks

Toolkit #30

zachsa opened this issue Mar 10, 2023 · 0 comments
Assignees
@zachsa @GilesFearon

Comments

@zachsa
Copy link
Collaborator

zachsa commented Mar 10, 2023 •
edited
Loading

  • Dockerized CLI should have default paths relative to the host /home/$USER directory. Otherwise the scripts download internally to the container by default
  • GFS download needs to be throttled to a maximum of 100 requests per second
  • Refactor GFS and OISST download to use the same functions
  • Add retry-on-fail decorator to async download functions
  • Convert to dynamic imports where possible as this will speed up the CLI execution and allow for better scalability

API

CROCO

toolkit croco -h

PostGIS

toolkit pg -h
zachsa added a commit that referenced this issue Apr 12, 2023
@zachsa
Refactored CLI layout to be more intuitive in terms of naming, and al…
029892f 
…so the module logic is no longer imported into weird places before executing (#30)
zachsa added a commit that referenced this issue Apr 19, 2023
@zachsa
Added top level 'croco' application to the CLI, and updated the CLI s…
52cf6b6 
…o that the __main__.py file doesn't have to be updated when adding cli options. Updated all modules to improve help output (#30)
zachsa added a commit that referenced this issue Apr 20, 2023
@zachsa
Updated toolkit download to have configurable hdays and fdays (#30)
b856f03
zachsa added a commit that referenced this issue Apr 20, 2023
@zachsa
Updated the toolkit to load postgis values in parallel (#30)
39767bc
zachsa added a commit that referenced this issue Apr 21, 2023
@zachsa
WIP updating toolkit CLI postgres operations to be better laid out (#30)
99b02af
zachsa added a commit that referenced this issue Apr 24, 2023
@zachsa
Re-organized toolkit CLI so that load operation into postgres doesn't…
607aa95 
… require any flags (all info comes from the netcdf file being processed), and refactored structure (#30)
zachsa added a commit that referenced this issue Apr 25, 2023
@zachsa
Refactored toolkit so that download is top level (#30), and adjusted …
cb91734 
…runs to use the new layout
zachsa added a commit that referenced this issue May 15, 2023
@zachsa
Updated toolkit layout - moved the load croco to pg code out of the c…
2b8ec70 
…roco directory (#30)
zachsa added a commit that referenced this issue May 15, 2023
@zachsa
Removed old pg client that doesn't support async (psycopg) and refact…
a5d86e8 
…ored all postgres related code to live within the pg cli application #30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@zachsa zachsa

@GilesFearon GilesFearon

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@zachsa @GilesFearon