Code

Add Icechunk Support#256

Code

Merged

TomNicholas merged 90 commits intozarr-developers:mainzarr-developers/VirtualiZarr:mainfrom mpiannucci:icechunkmpiannucci/VirtualiZarr:icechunkCopy head branch name to clipboardOct 22, 2024

+622-59

Commits

Commits on Sep 27, 2024

• move vds_with_manifest_arrays fixture up

  TomNicholas

  committed
  7b00e41

  Copy full SHA for 7b00e41

• sketch implementation

  TomNicholas

  committed
  c82221c

  Copy full SHA for c82221c

• test that we can create an icechunk store

  TomNicholas

  committed
  d29362b

  Copy full SHA for d29362b

• fixture to create icechunk filestore in temporary directory

  TomNicholas

  committed
  2aa3cb5

  Copy full SHA for 2aa3cb5

• get the async fixture working properly

  TomNicholas

  committed
  f2c095c

  Copy full SHA for f2c095c

• split into more functions

  TomNicholas

  committed
  6abe32d

  Copy full SHA for 6abe32d

• change mode

  TomNicholas

  committed
  93080b3

  Copy full SHA for 93080b3

• try creating zarr group and arrays explicitly

  TomNicholas

  committed
  bebf370

  Copy full SHA for bebf370

Commits on Sep 28, 2024

• create root group from store

  TomNicholas

  committed
  833e5f0

  Copy full SHA for 833e5f0

• todos

  TomNicholas

  committed
  9853140

  Copy full SHA for 9853140

• do away with the async pytest fixtures/functions

  TomNicholas

  committed
  030a96e

  Copy full SHA for 030a96e

• successfully writes root group attrs

  TomNicholas

  committed
  90ca5cf

  Copy full SHA for 90ca5cf

• check array metadata is correct

  TomNicholas

  committed
  b138dde

  Copy full SHA for b138dde

• try to write array attributes

  TomNicholas

  committed
  6631102

  Copy full SHA for 6631102

• sketch test for checking virtual references have been set correctly

  TomNicholas

  committed
  e92b56c

  Copy full SHA for e92b56c

Commits on Sep 30, 2024

• test setting single virtual ref

  TomNicholas

  committed
  2c8c0ee

  Copy full SHA for 2c8c0ee

• use async properly

  TomNicholas

  committed
  a2ce1ed

  Copy full SHA for a2ce1ed

Commits on Oct 1, 2024

• better separation of handling of loadable variables

  TomNicholas

  committed
  9393995

  Copy full SHA for 9393995

• fix chunk key format

  TomNicholas

  committed
  956e324

  Copy full SHA for 956e324

• use require_array

  TomNicholas

  committed
  2d7d5f6

  Copy full SHA for 2d7d5f6

• check that store supports writes

  TomNicholas

  committed
  8726e23

  Copy full SHA for 8726e23

• removed outdated note about awaiting

  TomNicholas

  committed
  387f345

  Copy full SHA for 387f345

Commits on Oct 2, 2024

• fix incorrect chunk key in test

  TomNicholas

  committed
  b2a0700

  Copy full SHA for b2a0700

• absolute path

  TomNicholas

  committed
  4ffb55e

  Copy full SHA for 4ffb55e

• convert to file URI before handing to icechunk

  TomNicholas

  committed
  f929fcb

  Copy full SHA for f929fcb

• test that without encoding we can definitely read one chunk

  TomNicholas

  committed
  e9c1287

  Copy full SHA for e9c1287

• Work on encoding test

  mpiannucci

  committed
  2fe3012

  Copy full SHA for 2fe3012

• Merge remote-tracking branch 'origin/icechunk' into matt/icechunk-encoding

  mpiannucci

  committed
  33d8ce8

  Copy full SHA for 33d8ce8

• Update test to match

  mpiannucci

  committed
  8aa6034

  Copy full SHA for 8aa6034

• Quick comment

  mpiannucci

  committed
  aa2d415

  Copy full SHA for aa2d415

• more comprehensive

  mpiannucci

  committed
  7e4e2ce

  Copy full SHA for 7e4e2ce

Commits on Oct 3, 2024

• add attrtirbute encoding

  mpiannucci

  committed
  9a03245

  Copy full SHA for 9a03245

Commits on Oct 4, 2024

• Merge pull request zarr-developers#2 from earth-mover/matt/icechunk-encoding

  Show description for 9676485
  TomNicholas

  authored
  9676485

  Copy full SHA for 9676485

Commits on Oct 10, 2024

• Fix array dimensions

  mpiannucci

  committed
  bbaf3ba

  Copy full SHA for bbaf3ba

Commits on Oct 11, 2024

• Merge pull request zarr-developers#3 from earth-mover/matt/array-dims

  mpiannucci

  authored
  31945cd

  Copy full SHA for 31945cd

• Fix v3 codec pipeline

  mpiannucci

  committed
  49daa7e

  Copy full SHA for 49daa7e

• Put xarray dep back

  mpiannucci

  committed
  756ff92

  Copy full SHA for 756ff92

Commits on Oct 12, 2024

• Handle codecs, but get bad results

  mpiannucci

  committed
  8c7242e

  Copy full SHA for 8c7242e

• Gzip an d zlib are not directly working

  mpiannucci

  committed
  666b676

  Copy full SHA for 666b676

Commits on Oct 13, 2024

• Get up working with numcodecs zarr 3 codecs

  mpiannucci

  committed
  9076ad7

  Copy full SHA for 9076ad7

Commits on Oct 14, 2024

• Update codec pipeline

  mpiannucci

  committed
  7a160fd

  Copy full SHA for 7a160fd

Commits on Oct 15, 2024

• Merge pull request zarr-developers#4 from earth-mover/matt/v3-codecs

  Show description for 286a9b5
  mpiannucci

  authored
  286a9b5

  Copy full SHA for 286a9b5

• oUdpate to latest icechunk using sync api

  mpiannucci

  committed
  8f1f96e

  Copy full SHA for 8f1f96e

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for b59060d
  pre-commit-ci[bot]

  committed
  b59060d

  Copy full SHA for b59060d

Commits on Oct 21, 2024

• Merge remote-tracking branch 'origin/main' into icechunk

  mpiannucci

  committed
  4f3bafa

  Copy full SHA for 4f3bafa

• Merge remote-tracking branch 'mpiannucci/icechunk' into icechunk

  mpiannucci

  committed
  26db575

  Copy full SHA for 26db575

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 01d261c
  pre-commit-ci[bot]

  committed
  01d261c

  Copy full SHA for 01d261c

• Some type stuff

  mpiannucci

  committed
  52e53f9

  Copy full SHA for 52e53f9

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for d10de6b
  pre-commit-ci[bot]

  committed
  d10de6b

  Copy full SHA for d10de6b

• Update zarr and icechunk tests, fix zarr v3 metadata

  mpiannucci

  committed
  d0b6bfb

  Copy full SHA for d0b6bfb

• Update import we dont need

  mpiannucci

  committed
  b7dc5f5

  Copy full SHA for b7dc5f5

• Update kerhcunk tests

  mpiannucci

  committed
  1e8ba7d

  Copy full SHA for 1e8ba7d

• Check for v3 metadata import in zarr test

  mpiannucci

  committed
  6b5b8b5

  Copy full SHA for 6b5b8b5

• More tests

  mpiannucci

  committed
  00ecae3

  Copy full SHA for 00ecae3

• type checker

  mpiannucci

  committed
  14ee6f9

  Copy full SHA for 14ee6f9

• types

  mpiannucci

  committed
  23d98de

  Copy full SHA for 23d98de

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for c633f13
  pre-commit-ci[bot]

  committed
  c633f13

  Copy full SHA for c633f13

• More types

  mpiannucci

  committed
  308a508

  Copy full SHA for 308a508

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 60cb43e
  pre-commit-ci[bot]

  committed
  60cb43e

  Copy full SHA for 60cb43e

• ooops

  mpiannucci

  committed
  d72e9d8

  Copy full SHA for d72e9d8

• Merge remote-tracking branch 'mpiannucci/icechunk' into icechunk

  mpiannucci

  committed
  0e0e5ac

  Copy full SHA for 0e0e5ac

• One left

  mpiannucci

  committed
  3873fde

  Copy full SHA for 3873fde

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 19bc9ae
  pre-commit-ci[bot]

  committed
  19bc9ae

  Copy full SHA for 19bc9ae

• Finally done being dumb

  mpiannucci

  committed
  4b6aefc

  Copy full SHA for 4b6aefc

• Merge remote-tracking branch 'mpiannucci/icechunk' into icechunk

  mpiannucci

  committed
  9fa9b38

  Copy full SHA for 9fa9b38

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 4d5c46a
  pre-commit-ci[bot]

  committed
  4d5c46a

  Copy full SHA for 4d5c46a

• Support loadables without tests

  mpiannucci

  committed
  5cb3d21

  Copy full SHA for 5cb3d21

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 3e31f21
  pre-commit-ci[bot]

  committed
  3e31f21

  Copy full SHA for 3e31f21

• Add test for multiple chunks to check order

  mpiannucci

  committed
  e105b78

  Copy full SHA for e105b78

Commits on Oct 22, 2024

• Add loadable varaible test

  mpiannucci

  committed
  ea52003

  Copy full SHA for ea52003

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 54d9bea
  pre-commit-ci[bot]

  committed
  54d9bea

  Copy full SHA for 54d9bea

• Add accessor, simple docs

  mpiannucci

  committed
  303e5cb

  Copy full SHA for 303e5cb

• Update icechunk.py

  Show description for 8838a1d
  

  mpiannucci

  and

  TomNicholas

  authored
  8838a1d

  Copy full SHA for 8838a1d

• Update accessor.py

  Show description for dd6c118
  

  mpiannucci

  and

  TomNicholas

  authored
  dd6c118

  Copy full SHA for dd6c118

• Fix attributes when loadables are available

  mpiannucci

  committed
  85de689

  Copy full SHA for 85de689

• Merge remote-tracking branch 'mpiannucci/icechunk' into icechunk

  mpiannucci

  committed
  9b471ec

  Copy full SHA for 9b471ec

• Protect zarr import

  mpiannucci

  committed
  8cd5237

  Copy full SHA for 8cd5237

• Fix import errors in icechunk writer

  mpiannucci

  committed
  542a953

  Copy full SHA for 542a953

• More protection

  mpiannucci

  committed
  305b0c6

  Copy full SHA for 305b0c6

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 2f8e270
  pre-commit-ci[bot]

  committed
  2f8e270

  Copy full SHA for 2f8e270

• i am bad at this

  mpiannucci

  committed
  7ce9f67

  Copy full SHA for 7ce9f67

• Add xarray roundtrip asserts

  mpiannucci

  committed
  8b4863e

  Copy full SHA for 8b4863e

• Add icechunk to api.rst

  mpiannucci

  committed
  b072535

  Copy full SHA for b072535

• Update virtualizarr/tests/test_writers/test_icechunk.py

  Show description for 45ae850
  

  mpiannucci

  and

  TomNicholas

  authored
  45ae850

  Copy full SHA for 45ae850

• More test improvements, update realeses.rst

  mpiannucci

  committed
  36eaad1

  Copy full SHA for 36eaad1

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 9f4e978
  pre-commit-ci[bot]

  committed
  9f4e978

  Copy full SHA for 9f4e978

• tmore testing

  mpiannucci

  committed
  3b496f5

  Copy full SHA for 3b496f5

• Merge remote-tracking branch 'mpiannucci/icechunk' into icechunk

  mpiannucci

  committed
  1e580a5

  Copy full SHA for 1e580a5

• Figure out tests for real this time

  mpiannucci

  committed
  117479c

  Copy full SHA for 117479c

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 46c41d0
  pre-commit-ci[bot]

  committed
  46c41d0

  Copy full SHA for 46c41d0