Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MFD flow direction producing nodata holes on super large rasters #350

Closed
phargogh opened this issue Sep 25, 2023 · 0 comments · Fixed by #351
Closed

MFD flow direction producing nodata holes on super large rasters #350

phargogh opened this issue Sep 25, 2023 · 0 comments · Fixed by #351
Assignees
@phargogh
Labels
bug Something isn't working in progress Working on it!

Comments

@phargogh
Copy link
Member

phargogh commented Sep 25, 2023 •
edited
Loading

For a NatCap project, we have a 30m DEM of Colombia that is resulting in disjointed streams. When a single watershed within Colombia is used instead, the streams are contiguous, as we expect.

I dug into this a bit more and the root cause appears to be that the flow direction routine is producing nodata holes where the streams are becoming disjointed, but only on the super large raster. I have the data.

I'd guess there's an integer overflow issue happening somewhere here because of the size of these rasters.
@phargogh phargogh added bug Something isn't working in progress Working on it! labels Sep 25, 2023
@phargogh phargogh self-assigned this Sep 25, 2023
phargogh added a commit to phargogh/pygeoprocessing that referenced this issue Sep 25, 2023
@phargogh
Making a bunch of signed integers in routing unsigned longs.
2e705ec 
This will avoid issues with extremely large rasters at the cost of
potentially double the memory usage.

RE:natcap#350
RE:natcap#246
phargogh added a commit to phargogh/pygeoprocessing that referenced this issue Sep 25, 2023
@phargogh
Allocating more memory to ints throughout routing.
4faf8d5 
RE:natcap#350
RE:natcap#246
@phargogh phargogh mentioned this issue Sep 25, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@phargogh phargogh

Labels
bug Something isn't working in progress Working on it!
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Improve int memory usage throughout routing phargogh/pygeoprocessing
1 participant
@phargogh