Skip to content

3.17
Compare
Choose a tag to compare
@alex-aizman alex-aizman released this 11 Apr 00:46
v1.3.17
094eb3d

Table of Contents

  • CLI v1.2
  • Python SDK v1.1.2
  • S3 compatibility and Botocore
  • API changes
  • Tests and Documentation
  • Core: bug fixes and improvements
  • Build and Continuous Integration
  • Extensions: Downloader, dSort, ETL

See also:

CLI

  • show all jobs !5645
  • start/stop job/xaction !5660
  • refresh rate and countdown; long-running 'show job' and friends !5651
  • 'show log node-name' to mimic 'tail -f' !5652, !5654
  • add custom duration flag and logic !5655
  • 'ais config (cluster|node|cli)', 'ais config reset', and friends !5656
  • bucket completions !5657
  • set-config to show all updates; tweak iter-fields reflection !5658
  • 'show job' to aggregate all categories and support all selections !5661
  • transition to using job display names (major) !5663
  • start, stop, show jobs and xactions (cont-d) !5665
  • amend and restructure jobs !5666
  • running xactions (completions) !5672
  • tweak config json printout; get-config from memory !5673
  • update backend config !5674
  • update backend config (part two) !5677
  • add footnote, marshal message only once !5678
  • remove xaction term and subcommand (everything is job now) !5692
  • suggest (targets, proxies, nodes) !5694, !5696
  • revise bash completions script !5697
  • remove 'xaction' (term and subcommand) !5698, !5699, !5700
  • 'show cluster': separate cluster nodes from all other (tab-tab) completions !5701
  • consolidate and refactor cluster map access !5704
  • tweak ais create bucket --props & ais bucket props set !5706
  • extend 'job start' to support (resilver, copy-bucket, rename-bucket) !5715
  • tweak listed props !5719
  • remove (cleanup) download and dsort jobs !5721
  • extend 'ais stop' to support --all|--regex !5722
  • 'show job' verbose option; unify usage args; ref PUT/APPEND !5723
  • rewrite command-not-found logic; add similar commands !5724
  • show jobs (major) !5726, !5727
  • bash autocomplete ordering improvements !5728
  • improvements (usability) !5729
  • add bucket cp alias !5730
  • flag printable name; split 'show job' in parts; usability !5736
  • further unify stopping, waiting-for, and showing jobs !5744
  • revise & amend 'show rebalance' - all permutations !5761
  • universal start-end formatting; template refactoring !5762
  • jobs grouping by name and, within name, by UUID !5764
  • complete etl-name transition !5767
  • ETL tools, UUID (part one) !5745, !5746, !5749, !5753, !5754, !5763
  • fix download/dsort progress !5769
  • new table to show target statistics !5788
  • 'ais show performance' (new) !5791, !5793, !5800, !5802, !5803, !5809, !5810, !5811, !5812, !5816
  • IEC, SI, and raw (bytes, nanoseconds) formatting (major) !5820
  • reduce code, simplify, cleanup !5821
  • IEC, SI, and raw (bytes, nanoseconds) formatting (major) !5823
  • disk stats: add average read/write sizes !5824
  • amend existing mountpath tab and add a new one !5833, !5834
  • expect node unreachable when iterating '--refresh' !5837
  • assorted usability; add 'no-color' config !5839
  • 'ais show performance': average (GET, PUT, etc.) sizes on the fly !5840
  • support new API to reset stats !5841, !5843
  • 'ais show performance': refactor throughput, add latency !5844
  • 'ais show performance': finalize latency tab !5847
  • 'ais show performance' cont-d !5848
  • 'ais show performance': finalize top-level tab !5850
  • 'ais show performance': add cluster-level throughput, beautify !5852
  • 'ais show performance': alias 'stats' and remove older code !5853
  • 'ais show performance': disk table v2 !5855
  • 'ais show performance': finalize disk table !5857
  • 'ais show performance': new mountpaths/disks/capacity table !5858, !5859
  • 'ais show performance': finalize capacity table !5861
  • refactor and cleanup multi-object put !5862
  • multi-object PUT: source dir, list/range; matching pattern !5865
  • fix concatenation logic, refactor progress bar !5867
  • copy bucket: support progress bars (copied objects and size) !5870
  • consistent timeout management !5871
  • copy/transform a list or range of objects: add progress bar !5873
  • copy/transform with progress bar: style, reuse !5874
  • multi-object PUT !5876
  • rogress bar: all multi-object operations; universal 'wait-for' !5879
  • PUT multi-object - all flavors !5880
  • get multiple objects in one shot (""multi-object GET"") !5884
  • GET destination & assorted fixes !5882
  • copy-bucket: prepend prefix, command helps, examples !5889, !5891
  • more inline help !5892
  • assorted improvements (minor) !5900
  • fix downloading with progress bar enabled !5903
  • how-to text: how to reconfigure remote ais cluster !5932
  • add CLI compatibility warning (new CLI vs old cluster and vice versa) !5952
  • cluster membership-changing operations (shutdown, decommission, et al): improve usability !5956

Python: SDK library and ETL

  • Add unit tests for cluster class !5675
  • Add unit test for api client class !5676
  • Fix python ETL test workflow !5680
  • Add unit tests for Xaction class !5681
  • Add unit test for object class, fix object put data from a filepath !5682
  • Add unit tests for ETL class !5683
  • Add unit tests for sdk utils !5685
  • Restructure python subdirectories and containment !5686
  • Run python unit tests as part of default python test make option, include python ETL tests in all python test runs !5688
  • Only run python ETL tests when python labels are added !5702
  • update test utils to support running Python tests on Windows !5716
  • Add multi-object functionality !5720
  • Add tests and validation to object ranges, support leading zeros !5731
  • Add unit test for object group class !5732
  • Update documentation for python multi-object ops !5734
  • Use aws bucket for python sdk CI tests to test caching functions !5739
  • Add string template support to object groups !5741
  • Refactor all references from xaction to job !5742
  • Update ETL runtime defaults and add new python 3.11 option !5743
  • Fix remote bucket tests to avoid collisions !5748
  • Increment cloudpickle version and update github action to use 3.11 for tests !5756
  • Fix python test dependencies !5780
  • Refactor to simplify typing and tests !5789
  • Bump python SDK to 1.0.5 !5790
  • Update python sdk version !5792
  • Update pytorch integration README with compatibility issue !5794
  • Patch sdk to support torchdata integration !5795
  • Bump aistore package version !5798
  • Fix pylint and formatting !5799
  • Add ObjAttr type for returning additional object metadata !5805
  • Address lint warnings, general improvements !5808
  • Add writer option to object get !5813
  • Split README for different python projects !5814
  • Add PROMOTE functionality to objects !5818
  • Standardize pylint version, fix lint errors !5822
  • Improve object put behavior and add directory put options !5826
  • Use Pathlib over os.path !5827
  • Refactor multi-file put to bucket class !5830
  • Improve job interface and update promote options !5832
  • Update python package build tools, increase version to 1.0.9 !5835
  • Improve example documentation !5842
  • Improve input validation !5846
  • Set up proper logging, update constants !5849
  • Add job wait for idle status and fix job bucket filter !5863
  • Add multi-object copy !5866
  • Fix remote test fixture !5877
  • Add multi-object ETL !5878
  • Release 1.0.10 !5881
  • Expand multi-object examples !5883
  • Improve usability !5888
  • Follow-up for copy/transform prepend !5890
  • Improve object interface !5893
  • Add prefix_filter option to bucket copy !5899
  • Add flags and target options to bucket list_objects methods !5901
  • Release 1.1.0 !5910
  • Release 1.1.1 !5911
  • Improve python sdk examples !5913
  • Add cluster list_jobs, require id for individual job status query !5916
  • Add support for multi-object archive !5920
  • Update bucket params to use Bucket object, support namespaces !5925
  • Add prefix filter to bucket transform !5949
  • Release 1.1.2 !5950
  • PyTorch: add support for etl-name in AISDataset and AISFileLoaderIterDataPipe !5957

S3 compatibility and Botocore

  • Add botocore monkey patch alongside python SDK !5684
  • Move botocore and pytorch packages to python top-level, separate from sdk !5691
  • Add s3 compatibility testing with boto3 !5703
  • list-objects vs HEAD !5705
  • compute multipart md5 and set etag !5708
  • Improve s3 compat test documentation and update validated tests !5747
  • Pass S3 delete a list of objects !5854
  • fix infinite loop when listing objects; add bucket name to list object response !5864
  • return bucket creation date in UTC !5869
  • new flag NoRecursion to support S3 delimiter feature !5930

API changes and new APIs

  • add API to query multiple xactions via any IC proxy !5670, !5671
  • yet another API to query xactions (new) !5687
  • list-objects API: tweak listed props !5718
  • [API change] flatten xaction-snap control structure (major upd) !5740
  • [API change] ETL: tools, UUID (part nine) !5754
  • [API change] ETL: new query parameter to specify transform name !5755, !5765
  • [API change] init/start ETL to return xaction ID !5768
  • [API change] GET(object) !5770, !5772, !5773, !5775
  • [API change] remove xaction 'query-msg' (deprecated) !5783, !5784, !5785
  • get-object API: amend comments !5769
  • refactor api package (major) !5776
  • [new API] query metric names and kinds ('counter', 'latency', 'throughput', et al.) !5796
  • [API change] get node status !5811
  • [API change] core stats: consistency between stats-querying APIs !5817
  • [API change] PUT(object) !5828
  • [API change] amend capacity-disks-filesystems control !5831
  • [API change] add API to reset stats !5841
  • [API change] copy bucket: add prefix option !5894
  • [API change] copy/transform bucket: add 'prefix' option !5895
  • [API change] bucket summary: add 'prefix' option (major) !5896, !5898
  • [API change] return comma-separated list of job UUIDs !5927
  • [API change] extend & refactor system info, add load averages !5951

Core

  • xactions: display name, global & bucket scope, abort by kind (core) !5662
  • xaction snaps V2 (major) !5667
  • tweak reversing-to-remais logic !5668
  • bucket summary !5678
  • bucket summary: handle failure to start !5679
  • offline access to remote ais cluster buckets !5674
  • refactor cluster.Target interface !5689
  • unify time formatting; log: current date periodically !5705
  • fix HEAD(remote-ais-bucket) when the latter is not present !5707
  • xaction registry: cleanup list-objects separately, revise house-keeping !5725
  • copy-bucket & transform-bucket: revise generic stats !5730
  • global rebalance: fix cleanup upon (re)transmit failures; handle err-stream-terminated !5733
  • extend xaction snap: add source and destination buckets !5735
  • always return atime and checksum when listing 'cached' content !5738
  • assorted fixes and ref (cont-d) !5759
  • add has-active-peers; refactoring !5760
  • special circumstance to retry delete-object; lesser assert !5782
  • consolidate xaction API messages !5786
  • rebalance upon cold reboot, refactor join/kalive !5787
  • GET(object): add version-changed stats counter !5801
  • core stats: lockless (counter, gauge, special) !5804
  • core stats: revise error counting (major) !5806, !5807
  • core stats: introduce KindSize !5815
  • IEC and IS units of measurement (ref) !5819
  • core: micro-optimizations and cleanup !5825
  • rename mountpath and filesystem (ref pass) !5829
  • stats: serialize to SGL !5834
  • stats: rename for consistency !5836
  • config features: add fsync-put; don't destroy bucket when creating one !5845
  • core: micro-optimize BMD update !5851
  • core: refactor 'ios' !5856
  • extend the minimal/default subset of list-objects queried props !5868
  • refactor packages: 'nl', 'api', 'xact' !5872
  • add 'idles-before-finishing' state !5875
  • parse IEC and SI units !5882
  • multi-object GET (part two) !5885
  • streaming-x: multi-bucket matching, transport name !5886
  • long timeout to query all xactions !5887
  • copy-bucket: prepend prefix !5889
  • LOM as a remote data provider (DP) !5905
  • copy/transform remote objects that may or may not be present (major) !5906
  • copy/transform remote objects that may or may not be present !5908
  • copy/transform remote objects (part three) !5909
  • end-to-end protection: compute a given checksum type only once !5912
  • copy/transform remote objects (part four) !5914
  • core/list-objects: always primary !5915
  • copy/transform remote objects (part four) !5917
  • canonical bucket/object name, and more !5918
  • copy/transform remote objects (part five); canonical names !5919, !5921
  • store cleanup: delay deleting bucket directories when in doubt !5923
  • copy/transform remote objects (part six) - tests !5924
  • remote ais cluster: fix API forwarding logic; transform/copy test !5926
  • when node is in maintenance don't try to query it's state-and-status !5956
  • revise waiting/polling logic, reset probe frequency !5960

Build and Continuous Integration (CI)

  • add workflow job to build ais-init image !5646
  • build: move CI (image, make); move/update testing-in-docker content !5710
  • build: upgrade all minors !5711
  • Update minikube deploy script, remove from test run !5774
  • Update docker/ci image to include docker io !5778
  • Add uuid dependency to docker/ci image !5779
  • build: clear golang/x/net security alerts (two commits) !5838
  • up modules; cleanup redundant err helpers, simplify; objhead !5907
  • lint with new linter !5929
  • upgrade linter !5931
  • build: remove 3rd party atomic package (use standard) !5934, !5935, !5936
  • math/rand vs crypto/rand ref !5937
  • build: transition to Go 1.20 !5938
  • up modules; ""end-to-end protection"" and other comments !5939
  • add k8s pipeline to run manually !5941, !5943

Tests and Documentation

  • tests: amend all-xaction-status; golangci-lint deadline !5781
  • rename 'ais/tests => ais/test' !5860
  • tests: fix copy/transform eviction logic, and misc !5928
  • extend ETL tests, add from/to permutations !5940
  • docs: 'show job' !5946
  • docs: 'ais job' !5947
  • docs: list objects and buckets, copy buckets, and more; add help.md !5953
  • tests: fix test-promote !5763
  • tests: remote bucket versioning !5766
  • main readme, cluster.md, storage.md, config.md updated, and more...

Extensions: Downloader, dSort, ETL

  • downloader: do not renew xaction !5645
  • amend and refactor dloader package (formerly, 'downloader') !5647, !5648, !5649
  • downloader and dsort: list-active !5650
  • downloader: cluster-wide xaction ID (major) !5659
  • downloader: continued refactoring !5660
  • move ais extensions: etl, downloader, dsort !5664
  • k8s: remove unused ExecCmd !5712
  • default bucket props vs cluster config !5713
  • dsort: add support for EKM with JSON structure !5717
  • ETL: tools, UUID (part one) !5745
  • ETL: tools, UUID (part two) !5746
  • ETL: tools, UUID (parts three & four) !5749
  • ETL: CLI, tools, UUID (parts three & four) !5751
  • ETL: use (logical) "etl_name" instead of (unique) "etl_id" and "uuid" (major update) !5752
  • ETL: tools, UUID (part eight) !5753
  • Add pod health API endpoint for ETL !5771
  • ETL: assorted renames for consistency !5777
  • ETL stats; xaction base notif abstraction (minor) !5902
  • downloader: reduce "wrapping" and deferring !5904
Assets 8
Loading