Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow data to specify statistics about the data to prevent it having to be computed #2133

Open
Cadair opened this issue May 7, 2020 · 4 comments

Comments

@Cadair
Copy link
Contributor

Cadair commented May 7, 2020

Is your feature request related to a problem? Please describe it:
When using large distributed datasets, or data on slower storage devices it would be useful to be able to use precomputed statistics about the data from the metadata in the file.

This could include things like min / max / std / 99.9% etc. This would then be used by the image viewer etc rather than sampling the array to compute these things.

In addition it might be useful if these statistics could be specified per-axis or per dask chunk. This would allow axes like Stokes to have different dynamic ranges over the whole axis without having to do the computation (which could lead to a lot of dask chunks being loaded).

Describe the solution you'd like:
An API for data factories to be able to specify these precomputed statistics.

@astrofrog
Copy link
Member

Just a quick note that for more complex cases, overloading compute_statistic might be the easiest way

@kakirastern
Copy link

Am really interested in working on this issue.

@kakirastern
Copy link

Let me start with some AIA and HMI data as @Cadair has suggested first.

@kakirastern
Copy link

Found the keywords needed for the statistics of AIA and HMI data sets. Let me try to do the same for IRIS Level 2 raster and SJI data cubes to see if something similar can be found.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants