Releases v1.3.18
Core
add htext to track restarted state; target run and misc !5966
cluster rebalance (scenarios) !5969, !5971, !5973, !5974, !5975, !5977, !5980, !5983, !5986, !5987, !5989, !5991, !5992, !5993, !5995, !6002
add 'cluster-ready' helper; use it to reinforce !5976
cleanup better when decommissioning; previous BMD at startup !5979
fs: reliable remove-all !5981, !5982, !5984
yet another buf pool !5985
do not modify cluster map when starting up; always skip logging idle disks !5988
rebalance (scenarios, major update) !5992
[API change]: core: rebalance (scenarios) !5993
rebalance (major update); when receiving new cluster map !5995
up modules; handle housekeeper registration race !5994
'not present in the loaded cluster map' and similar startup validation !5996
shutdown or decommission a node that's already in maintenance !5998
transport: never establish a streaming connection to the peer that's in maintenance (or will be) !5999
metasync just-in-time; assorted refactoring (minor) !6001maintenance mode: pre & post vs keepalive & metasync; CLI: more colored cues !6004
shutdown is also 'maintenance'; docs: adding-removing intro !6005
add meta package !6006, !6007
ETL: add arg-type parameter when initializing with code !6008
archive v2: support empty template (tar entire bucket); atime !6013
keep poi.atime in nanoseconds !6015
archive v2: append to arch; refactoring !6017
archive v2: up modules !6018
archive v2: part four (major) !6019
archive v2: detect an empty tar when appending, and handle !6020
archive v2: part six !6022
archive v2: mime detection !6024
archive v2: extend 'append-to-arch' to support tar.gz !6025, !6027
archive v2: tar and tgz append; fixes !6028
log filenames; overlapping run vs node-restarted !6029
archive v2: multi-object append-to-arch !6030
archive v2: multi-object append-to-arch !6033
archive v2: multi-object append-to-arch !6034
cleanup disk utils (minor) !6035
ios startup: run the command only once !6036
hide AuthN secret !6038
archive v2: append to zip !6041
archive v2: append to msgpack !6043
add cmn/archive package !6044
archive v2: write and copy via new 'cmn/archive' !6045
archive v2: append via new 'cmn/archive' !6046
[API change] archive v2: MIME vs file extensions !6047
ios: cleanup lsblk cache; CLI: refactor get-node-arg; up modules !6048
archive v2: remove msgpack; refactor !6051
archive v2: add '.tar.lz4' serialization (new) !6053
archive v2: tar.lz4 cont-d !6054
archive v2: lz4 features; checksum !6055
s3 compat: run E2E tests with correct HTTP/HTTPS mode !6057
[API change]: append to arch if exists !6062
[API change] append to arch if doesn't exist; CLI cont-d !6064
checksumming and buffering vs reader-from !6074
core: content-length universally; revise write-json and friends !6075, !6076
archive v2: [API change] put (files, dirs) with an option to append !6081, !6082
archive v2: quiesce faster, refine continue-on-error logic !6083
core: double-check target-in-maintenance, quiesce faster !6084
archive v2: finalize cmn/archive package !6085
archive v2: finalize cmn/archive package !6086
log verbosity: core and modules !6087
http client: disable compression; core: undefer & micro-optimize !6066
append to (non-existing) arch: an option to create !6068
mem-pool alloc/free symmetry: copy/transform & archive !6069
copy/transform, multi-archive: refactor Rx logic and error handling !6071
log verbosity: core and modules; remote cluster !6089
log verbosity: core and modules; remote cluster !6091
ec: minor refactoring !6092
archive v2: WD basename; get with extraction; Range !6093
archive v2: WD basename; get with extraction; Range !6094
archive v2: tools/archive utils !6095
archive v2: tools/archive utils !6096
compile-out asserts; super-verbose logging; log module 'mirror' (ref) !6097
log verbosity at runtime; log modules; remove glog; unify (major update) !6099
fields iterator; size converter; log rotation (fixes) !6101
[API change] get-bucket-info to count remote objects !6102
[API change] get-bucket-info to count remote objects !6103
[API change] get-bucket-info (part three); docs and CLI !6106
list-objects vs buckets: revise and refactor, add validation, clarify !6107
list-objects: introduce optional args (ref, cleanup) !6108
list-objects: mem-pool msgpack buffers !6109
kvdb: remove redundant err-not-found; amend dsort, downloader, authn !6110
x-lso must idle more time !6111
log modules (part three) !6112
add nlog (new logger) !6113, !6122, !6124
do not log perf counters when there's no change; sort the names !6119
fix disk usage call for clusters on mac OS !6121
log etl events: spec parsed, pod ready, hpull/hpush !6125
cleanup fs.PathError; add object name validation !6127
extend dsort to support .tar.lz4 !6128
aisloader: cleanup output when running with json option !6129
xactions to return extended error info !6130
xactions: add and return errors (major upd) !6131, !6132
xactions: error handling cont-d; bucket-scope multi-object op-s !6133
jobs error handling and reporting; tests !6134, !6135, !6136
use Go 1.20 join-err; reduce default pre-election interval !6141
http request multiplexer !6142
dsort: use common cmn/archive pkg (major upd) !6143
archive v2: tar formats (USTAR, etc.) !6153
archive v2: extend list-objects and GET to operate inside archives !6155
archive v2: extend list-objects and GET to operate inside archives !6157
control plane: consistently propagate cluster map !6158
old EC metadata shouldn't terminate cluster-wide rebalance !6159
core: when primary goes down it notifies !6160
CLI
performance tabs: always show 'cluster idle' if idle !5997
stop-maintenance w/ no rebalance; stats idle-ness (minor) !6000
add transform-url flag (used when initializing ETL with code) !6010
misc. improvements !6011
mountpath completions; disable/detach; minor ref !6056de-spaghettify put handler !6059
archive multi-object (cont-d) !6060
archive v2: CLI put, append, alias, docs !6072
move 'gen-shards' and extend it to support all formats !6073
archive v2: CLI: APPEND-to-arch is now on-par with multi-PUT !6077
archive v2: CLI: dry-run option is back !6078
archive v2: CLI: destination naming; dry-run; tips and examples !6079
list-objects: extended help, template with no ranges !6104
error and warning verbosity (major update) !6105
log modules (part two); CLI archive: pre-parse and add tip, advice !6100
archive help !6126
archive v2: CLI and docs !6149
archive v2: multiple CLI updates and improvements; is-archive bit !6150
remove k8s apimachinery pkg (minor) !6151
Python
sdk/python: Add option to pre-import modules when initializing ETL with code !5963
sdk/python: Refactor ETL to provide name once on object creation !5968
sdk/python: Release version 1.2.0 !5970
sdk/python: Fix python package links !5978
sdk/python: improve SDK testing, fix bucket eviction keep_md parameter !5990
sdk/python: Release version 1.2.2 !6012
sdk/python: Add dSort support in SDK !6032
sdk/python: Add wait to dsort abort test !6037
sdk/python: Fix dsort abort test on faster systems !6039
sdk/python: Add get_url option to bucket, object group, and objects !6042
sdk/python: Add interface for AIS sources that can be accessed via list of URLs !6052
PyTorch: Add new PyTorch DataPipe to iterate over URLs from various AIS sources !6058
update lint-tests to work with Python 3.11 (pytorch unsupported) !6061
sdk/python: Include user agent in SDK requests !6067
sdk/python Use msgpack content type when listing objects in bucket !6070
sdk/python: Add msgspec to pyproject dependencies !6088
sdk/python: Release SDK version 1.3.0 !6090
sdk/python: bucket summary + bucket info !6098
sdk/python: bck info integration test fix !6115
sdk/python: bck summary integration test fix !6117
sdk/python: bck summ + bck info fixes (revert and cleanup) !6120
sdk/python: pyaisloader !6123
pyaisloader: bucket utils (minor fixes) !6139
sdk/python: bucket summary and info (minor fixes) !6140
pyaisloader: total size/count fix !6146sdk/python: Release version 1.4.0 !6152
Docs
add lifecycle.md !6009
WebDataset Blog Post part 1: Storing WebDataset in AIS !6014
WebDataset Blog Post part 2: AIS ETL on WebDataset shards !6016
update docs/tools and cmd/README !6050
WebDataset blog post pt 3 -- PyTorch Datatpipe with ETL !6065archive v2: CLI put, append, alias, docs !6072
archive v2: CLI: destination naming; dry-run; tips and examples !6079
archive v2: CLI and docs !6149
Build, CI
add local build options to k8s scripts and fix local registry !6021
build: upgrade all minors !6040
add pyaisloader stage !6148
k8s: fix deployment scripts for compatibility with the latest aisnode image !6154
You can’t perform that action at this time.
