Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

performance improvement - separate out raster generation for each waterway #262

Closed
rosepearson opened this issue Oct 6, 2024 · 2 comments · Fixed by #264
Closed

performance improvement - separate out raster generation for each waterway #262

rosepearson opened this issue Oct 6, 2024 · 2 comments · Fixed by #264
Milestone

Comments

@rosepearson
Copy link
Owner

rosepearson commented Oct 6, 2024

Currently we produce a single waterway raster for all waterways considered together. This leads to errors with large files. consider generating each separately as generally these will be much smaller than the overall waterway network. This should simplify memory management and reduce errors with files failing to be saved.

Actions:

  • Save each waterway separately
  • Save each stopbank separately
@rosepearson
Copy link
Owner Author

rosepearson commented Oct 9, 2024

Note failing CI seems related to issues specific to netcdf4 1.7.1 on https://github.com/Unidata/netcdf4-python/issues?q=is%3Aissue%20state%3Aopen%20hdf%20error

This si something @jennan ran into recently as referenced in this commit - https://git.niwa.local/forecasting-services/packages/highres_nwm/-/commit/f45845f7a19739f73c94d920e6acd86301d60020 he migrated to h5netcdf and also builds his environments using pip.

@rosepearson
Copy link
Owner Author

Further notes on CI errors related to saving / loading rasters:

Comparing the last successful CI tests: https://github.com/rosepearson/GeoFabrics/actions/runs/10825077248/job/30033393178 against the first failing tests: https://github.com/rosepearson/GeoFabrics/actions/runs/10839705758/job/31267478081

Library versions (All conda installed):
netcdf4 - 1.7.1 - same
libnetcdf - 4.9.2 - same
libgdal-hdf5 - 3.9.2 - same
libgdal-netcdf - 3.9.2 - same
libgdal-hdf4 - 3.9.2 - same
hdf4 - 4.2.15 - same
hdf4 - 1.14.3 - same
rioxarray - 0.17.0 same
rasterio - 1.3.11 - same

@rosepearson rosepearson added this to the 1.1.21 milestone Oct 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant