Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update roadmap documentation #398

Merged
merged 1 commit into from
Nov 19, 2024
Merged

Update roadmap documentation #398

merged 1 commit into from
Nov 19, 2024

Conversation

paraseba
Copy link
Collaborator

Split between pre and post 1.0 roadmaps.

@paraseba paraseba requested a review from rabernat November 19, 2024 16:01
Copy link
Contributor

@rabernat rabernat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great Seba!

Where appropriate, please link explicitly to existing issues / PRs.

docs/docs/contributing.md Outdated Show resolved Hide resolved
- Support GCP
- Distributed write support with dask.array
- Credential sets for virtual datasets
- Complete Python API
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we make this more specific?

- Commit conflict detection, resolution and rebase
- Current session status (git status)
- Support GCP
- Distributed write support with dask.array
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure we want to make changes in Dask required for 1.0. That introduces a huge uncertainty in the timeline. I think @dcherian's solution in #357 basically solves the problem. I suppose we would need an equivalent version for plain Dask arrays, but that can live in Icechunk, not Dask.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting, I thought Deepak's was the ultimate solution we needed.


#### Performance

- Create scale benchmark of daily updated dataset (15M chunks)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should also add a target number of commits. Maybe base this on Sylvera's repos.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Going with 30k commits


#### Refactoring

- Improve Python API
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Too vague.

Split between pre and post 1.0 roadmaps.

Update docs/docs/contributing.md

Co-authored-by: Ryan Abernathey <[email protected]>
PR feedback
@paraseba paraseba merged commit ee982e7 into main Nov 19, 2024
1 check passed
@paraseba paraseba deleted the push-qvmqylxwwurk branch November 19, 2024 16:50
dcherian added a commit that referenced this pull request Nov 22, 2024
* main:
  Set store to read only after unpickling (#405)
  Update roadmap documentation (#398)
  Wire up Zarr property tests EAR-1189 (#68)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants