From c5837d4aace6e4255bdd9dcc41f40a9aa93dba66 Mon Sep 17 00:00:00 2001 From: Nikolay Perfilov Date: Tue, 26 Nov 2024 16:44:32 +0300 Subject: [PATCH] Add ydb cli changelog (#11986) Co-authored-by: Aleksey Myasnikov --- ydb/apps/ydb/CHANGELOG.md | 258 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 258 insertions(+) create mode 100644 ydb/apps/ydb/CHANGELOG.md diff --git a/ydb/apps/ydb/CHANGELOG.md b/ydb/apps/ydb/CHANGELOG.md new file mode 100644 index 000000000000..c766b4663376 --- /dev/null +++ b/ydb/apps/ydb/CHANGELOG.md @@ -0,0 +1,258 @@ +* Improved throughput of `ydb import file csv` command. It is now approximately x3 times faster +* Allow running stock bench for `OLAP` shards +* Specify more clearly what concrete type of timestamp is in options in `ydb topic` commands +* Added support different timestamp formats in `ydb topic` commands +* Added `--explain-ast` option to `ydb sql` command that prints query AST +* Highlighting in interactive mode switched to common lexer using ANSI SQL syntax so ANSI queries can be highlighted correctly +* Added location printing for errors in `ydb tools restore` command +* Diffs in `ydb workload` benchmarks are now printed prettier +* Fixed progress bar in `ydb workload import` command +* Added pg syntax to tpch and tpcds benchmarks +* Fixed a bug where restoring from a backup using --import-data could fail if the partitioning of the table was changed +* In the `ydb topic write` command the `--codec` option now has default value `RAW`. +* Added log events for `ydb tools dump` and `ydb tools restore` commands +* Added `-c` option for `ydb workload tpcds run` command to compare the result with expected value and show the diff + +## 2.15.0 ## + +* Description is not ready yet + +## 2.14.0 ## + +* Description is not ready yet + +## 2.13.0 ## + +* Description is not ready yet + +## 2.12.0 ## + +* Description is not ready yet + +## 2.11.0 ## + +* Description is not ready yet + +## 2.10.0 ## + +### Features + +* Added the `ydb sql` command that runs over QueryService and can execute any DML/DDL command. +* Added `notx` support for the `--tx-mode` option in `ydb table query execute`. +* Added start and end times for long-running operation descriptions (export, import). +* Added replication description support in the `ydb scheme describe` and `ydb scheme ls` commands. +* Added big datetime types support: `Date32`, `Datetime64`, `Timestamp64`, `Interval64`. +* `ydb workload` commands rework: + + * Added the `--clear` option to the `init` subcommand, allowing tables from previous runs to be removed before workload initialization. + * Added the `ydb workload * import` command to prepopulate tables with initial content before executing benchmarks. + +### Backward incompatible changes + +* `ydb workload` commands rework: + + * The `--path` option was moved to a specific workload level. For example: `ydb workload tpch --path some/tables/path init ...`. + * The `--store=s3` option was changed to `--store=external-s3` in the `init` subcommand. + + +### Bug fixes + +* Fixed colors in the `PrettyTable` format + +## 2.9.0 ## + +### Features + +* Improved query logical plan tables: added colors, more information, fixed some bugs. +* The verbose option `-v` is supported for `ydb workload` commands to provide debug information. +* Added an option to run `ydb workload tpch` with an S3 source to measure [federated queries](concepts/federated_query/index.md) performance. +* Added the `--rate` option for `ydb workload` commands to control the transactions (or requests) per second limit. +* Added the `--use-virtual-addressing` option for S3 import/export, allowing the switch to [virtual hosting of buckets](https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html) for the S3 path layout. +* Improved `ydb scheme ls` performance due to listing directories in parallel. + +### Bug fixes + +* Resolved an issue where extra characters were truncated during line transfers in CLI tables. +* Fixed invalid memory access in `tools restore`. +* Fixed the issue of the `--timeout` option being ignored in generic and scan queries, as well as in the import command. +* Added a 60-second timeout to version checks and CLI binary downloads to prevent infinite waiting. +* Minor bug fixes. + +## 2.8.0 ## + +### Features + +* Added configuration management commands for the cluster `ydb admin config` and `ydb admin volatile-config`. +* Added support for loading PostgreSQL-compatible data types by [ydb import file csv|tsv|json](reference/ydb-cli/export-import/import-file.md) command. Only for row-oriented tables. +* Added support for directory load from an S3-compatible storage in the [ydb import s3](reference/ydb-cli/export-import/import-s3.md) command. Currently only available on Linux and Mac OS. +* Added support for outputting the results of [ydb table query execute](reference/ydb-cli/table-query-execute.md), [ydb yql](reference/ydb-cli/yql.md) and [ydb scripting yql](reference/ydb-cli/scripting-yql.md) commands in the [Apache Parquet](https://parquet.apache.org/docs/) format. +* In the [ydb workload](reference/ydb-cli/commands/workload/index.md) commands, the `--executer` option has been added, which allows to specify which type of queries to use. +* Added a column with median benchmark execution time in the statistics table of the [ydb workload clickbench](reference/ydb-cli/workload-click-bench.md) command. +* **_(Experimental)_** Added the `generic` request type to the [ydb table query execute](reference/ydb-cli/table-query-execute.md) command, allowing to perform [DDL](https://en.wikipedia.org/wiki/Data_Definition_Language) and [DML](https://en.wikipedia.org/wiki/Data_Manipulation_Language) operations, return with arbitrarily-sized results and support for [MVCC](concepts/mvcc.md). The command uses an experimental API, compatibility is not guaranteed. +* **_(Experimental)_** In the `ydb table query explain` command, the `--collect-diagnostics` option has been added to collect query diagnostics and save it to a file. The command uses an experimental API, compatibility is not guaranteed. + +### Bug fixes + +* Fixed an error displaying tables in `pretty` format with [Unicode](https://en.wikipedia.org/wiki/Unicode) characters. + +* Fixed an error substituting the wrong primary key in the command [ydb tools pg-convert](postgresql/import.md#pg-convert). + +## 2.7.0 ## + +### Features + +* Added the [ydb tools pg-convert](postgresql/import.md#pg-convert) command, which prepares a dump obtained by the [pg_dump](https://www.postgresql.org/docs/current/app-pgdump.html) utility for loading into the YDB postgres-compatible layer. +* Added the `ydb workload query` load testing command, which loads the database with [script execution queries](reference/ydb-cli/yql.md) in multiple threads. +* Added a command `ydb scheme permissions list` to list permissions. +* In the commands [ydb table query execute](reference/ydb-cli/table-query-execute.md), [ydb table query explain](reference/ydb-cli/commands/explain-plan.md), [ydb yql](reference/ydb-cli/yql.md), and [ydb scripting yql](reference/ydb-cli/scripting-yql.md), the `--flame-graph` option has been added, specifying the path to the file in which you need to save the visualization of query execution statistics. +* [Special commands](reference/ydb-cli/interactive-cli.md#spec-commands) in the interactive query execution mode are now case-insensitive. +* Added validation for [special commands](reference/ydb-cli/interactive-cli.md#spec-commands) and their [parameters](reference/ydb-cli/interactive-cli.md#internal-vars). +* Added table reading in the scenario with transactions in the command [ydb workload transfer topic-to-table run](reference/ydb-cli/workload-transfer.md#run). +* Added the `--commit-messages` option to the command [ydb workload transfer topic-to-table run](reference/ydb-cli/workload-transfer.md#run), specifying the number of messages in a single transaction. +* Added the options `--only-table-in-tx` and `--only-topic-in-tx` in the command [ydb workload transfer topic-to-table run](reference/ydb-cli/workload-transfer.md#run), specifying restrictions on the types of queries in a single transaction. +* Added new columns `Select time` and `Upsert time` in the statistics table in the command [ydb workload transfer topic-to-table run](reference/ydb-cli/workload-transfer.md#run). + +### Bug fixes + +* Fixed an error when loading an empty JSON list by commands: [ydb table query execute](reference/ydb-cli/table-query-execute.md), [ydb scripting yql](reference/ydb-cli/scripting-yql.md) and [ydb yql](reference/ydb-cli/yql.md). + +## 2.6.0 ## + +### Features + +* Added `--path` option to [ydb workload tpch run](reference/ydb-cli/workload-tpch.md#run), which contains the path to the directory with tables created by the [ydb workload tpch init](reference/ydb-cli/workload-tpch.md#init) command. +* Added [ydb workload transfer topic-to-table run](reference/ydb-cli/workload-transfer.md) command, which loads the database with read requests from topics and write requests to the table. +* Added the option `--consumer-prefix` in the commands [ydb workload topic init](reference/ydb-cli/workload-topic.md#init), [ydb workload topic run read|full](reference/ydb-cli/workload-topic.md#run-read), specifying prefixes of consumer names. +* Added the `--partition-ids` option in the [ydb topic read](reference/ydb-cli/topic-read.md) command, which specifies a comma-separated list of topic partition identifiers to read from. +* Added support for CSV and TSV parameter formats in [YQL query](reference/ydb-cli/parameterized-queries-cli.md) execution commands. +* The [interactive mode of query execution](reference/ydb-cli/interactive-cli.md) has been redesigned. Added [new interactive mode specific commands](reference/ydb-cli/interactive-cli.md#spec-commands): `SET`, `EXPLAIN`, `EXPLAIN AST`. Added saving history between CLI launches and auto-completion of YQL queries. +* Added the command [ydb config info](reference/ydb-cli/commands/config-info.md), which outputs the current connection parameters without connecting to the database. +* Added the command [ydb workload kv run mixed](reference/ydb-cli/workload-kv.md#mixed-kv), which loads the database with write and read requests. +* The `--percentile` option in the [ydb workload topic run write|read|full](reference/ydb-cli/workload-topic.md#run-write) commands can now take floating point values. +* The default values for the `--seconds` and `--warmup` options in the [ydb workload topic run write|read|full](reference/ydb-cli/workload-topic.md#run-write) commands have been increased to 60 seconds and 5 seconds, respectively. +* Changed the default value for the `--supported-codecs` option to `RAW` in the [ydb topic create](reference/ydb-cli/topic-create.md) and [ydb topic consumer add](reference/ydb-cli/topic-consumer-add.md) commands. + +### Bug fixes + +* Fixed string loss when loading with the [ydb import file json](reference/ydb-cli/export-import/import-file.md) command. +* Fixed ignored statistics during the warm-up of commands [ydb workload topic run write|read|full](reference/ydb-cli/workload-topic.md#run-write). +* Fixed incomplete statistics output in the [ydb scripting yql](reference/ydb-cli/scripting-yql.md) and [ydb yql](reference/ydb-cli/yql.md) commands. +* Fixed incorrect output of progress bar in [ydb tools dump](reference/ydb-cli/export-import/tools-dump.md) and [ydb tools restore](reference/ydb-cli/export-import/tools-restore.md) commands. +* Fixed loading large files with the header in the [ydb import file csv|tsv](reference/ydb-cli/export-import/import-file.md) command. +* Fixed hanging of the [ydb tools restore --import-data](reference/ydb-cli/export-import/tools-restore.md#optional) command. +* Fixed error `Unknown value Rejected` when executing the [ydb operation list build index](reference/ydb-cli/operation-list.md) command. + +## 2.5.0 ## + +### Features + +* For the `ydb import file` command, a parameter [--timeout](reference/ydb-cli/export-import/import-file.md#optional) has been added that specifies the time within which the operation should be performed on the server. +* Added a progress bar in commands [ydb scheme rmdir --recursive](reference/ydb-cli/commands/dir.md#rmdir) and [ydb import file](reference/ydb-cli/export-import/import-file.md). +* Added the command [ydb workload kv run read-rows](reference/ydb-cli/workload-kv.md#read-rows-kv), which loads the database with requests to read rows using a new experimental API call ReadRows (implemented only in the [main](https://github.com/ydb-platform/ydb) branch), which performs faster key reading than [select](reference/ydb-cli/workload-kv.md#select-kv). +* New parameters `--warmup-time`, `--percentile`, `--topic` have been added to the [ydb workload topic](reference/ydb-cli/workload-topic.md), setting the test warm-up time, the percentile in the statistics output and the topic name, respectively. +* Added the [ydb workload tpch](reference/ydb-cli/workload-tpch.md) command to run the TPC-H benchmark. +* Added the `--ordered` flag in the command [ydb tools dump](reference/ydb-cli/export-import/tools-dump.md), which preserves the order by primary key in tables. + +### Performance + +* The data loading speed in the `ydb import file` command has been increased by adding parallel loading. The number of threads is set by the new parameter [--threads](reference/ydb-cli/export-import/import-file.md#optional). +* A performance of the [ydb import file json](reference/ydb-cli/export-import/import-file.md) command has been increased by reducing the number of data copies. + +## 2.4.0 ## + +### Features + +* Added the ability to upload multiple files in parallel with the command [ydb import file](reference/ydb-cli/export-import/import-file.md#multiple-files). +* Added support for deleting column tables for the command [ydb scheme rmdir --recursive](reference/ydb-cli/commands/dir.md#rmdir). +* Improved stability of the command [ydb workload topic](reference/ydb-cli/workload-topic.md). + +## 2.3.0 ## + +### Features + +* Added the interactive mode of query execution. To switch to the interactive mode, run [ydb yql](reference/ydb-cli/yql.md) without arguments. This mode is experimental: backward compatibility is not guaranteed yet. +* Added the [ydb index rename](reference/ydb-cli/commands/secondary_index.md#rename) command for [atomic replacement](dev/secondary-indexes.md#atomic-index-replacement) or renaming of a secondary index. +* Added the `ydb workload topic` command for generating the load that reads messages from topics and writes messages to topics. +* Added the [--recursive](reference/ydb-cli/commands/dir.md#rmdir-options) option for the `ydb scheme rmdir` command. Use it to delete a directory recursively, with all its content. +* Added support for the `topic` and `coordination node` types in the [ydb scheme describe](reference/ydb-cli/commands/scheme-describe.md) command. +* Added the [--commit](reference/ydb-cli/topic-read.md#osnovnye-opcionalnye-parametry) option for the `ydb topic consumer` command. Use it to commit messages you have read. +* Added the [--columns](reference/ydb-cli/export-import/import-file.md#optional) option for the `ydb import file csv|tsv` command. Use it as an alternative to the file header when specifying a column list. +* Added the [--newline-delimited](reference/ydb-cli/export-import/import-file.md#optional) option for the `ydb import file csv|tsv` command. Use it to make sure that your data is newline-free. This option streamlines import by reading data from several file sections in parallel. + +### Bug fixes + +* Fixed the bug that resulted in excessive memory and CPU utilization when executing the `ydb import file` command. + +## 2.2.0 ## + +### Features + +* Fixed the error that didn't allow specifying supported compression algorithms when adding a topic consumer. +* Added support for streaming YQL scripts and queries based on options [transferred via `stdin`](reference/ydb-cli/parameterized-queries-cli.md). +* You can now [use a file](reference/ydb-cli/parameterized-queries-cli.md) to provide YQL query options +* Password input requests are now output to `stderr` instead of `stdout`. +* You can now save the root CA certificate path in a [profile](reference/ydb-cli/profile/index.md). +* Added a global option named [--profile-file](reference/ydb-cli/commands/global-options.md#service-options) to use the specified file as storage for profile settings. +* Added a new type of load testing: [ydb workload clickbench](reference/ydb-cli/workload-click-bench). + +## 2.1.1 ## + +### Improvements + +* Added support for the `--stats` option of the [ydb scheme describe](reference/ydb-cli/commands/scheme-describe.md) command for column-oriented tables. +* Added support for Parquet files to enable their import with the [ydb import](reference/ydb-cli/export-import/import-file.md) command. +* Added support for additional logging and retries for the [ydb import](reference/ydb-cli/export-import/import-file.md) command. + +## 2.1.0 ## + +### Features + +* You can now [create a profile non-interactively](reference/ydb-cli/profile/create.md#cmdline). +* Added the [ydb config profile update](reference/ydb-cli/profile/create.md#update) and [ydb config profile replace](reference/ydb-cli/profile/create.md#replace) commands to update and replace profiles, respectively. +* Added the `-1` option for the [ydb scheme ls](reference/ydb-cli/commands/scheme-ls.md) command to enable output of a single object per row. +* You can now save the IAM service URL in a profile. +* Added support for username and password-based authentication without specifying the password. +* Added support for AWS profiles in the [ydb export s3](reference/ydb-cli/export-import/auth-s3.md#auth) command. +* You can now create profiles using `stdin`. For example, you can pass the [YC CLI](https://cloud.yandex.ru/docs/cli/) `yc ydb database get information` command output to the `ydb config profile create` command input. + +### Bug fixes + +* Fixed the error when request results were output in JSON-array format incorrectly if they included multiple server responses. +* Fixed the error that disabled profile updates so that an incorrect profile was used. + +## 2.0.0 ## + +### Features + +* Added the ability to work with topics: + + * `ydb topic create`: Create a topic. + * `ydb topic alter`: Update a topic. + * `ydb topic write`: Write data to a topic. + * `ydb topic read`: Read data from a topic. + * `ydb topic drop`: Delete a topic. + +* Added a new type of load testing: + + * `ydb workload kv init`: Create a table for kv load testing. + * `ydb workload kv run`: Apply one of three types of load: run multiple `UPSERT` sessions, run multiple `INSERT` sessions, or run multiple sessions of GET requests by primary key. + * `ydb workload kv clean`: Delete a test table. + +* Added the ability to disable current active profile (see the `ydb config profile deactivate` command). +* Added the ability to delete a profile non-interactively with no commit (see the `--force` option under the `ydb config profile remove` command). +* Added CDC support for the `ydb scheme describe` command. +* Added the ability to view the current DB status (see the `ydb monitoring healthcheck` command). +* Added the ability to view authentication information (token) to be sent with DB queries under the current authentication settings (see the `ydb auth get-token` command). +* Added the ability for the `ydb import` command to read data from stdin. +* Added the ability to import data in JSON format from a file or stdin (see the `ydb import file json` command). + +### Improvements + +* Improved command processing. Improved the accuracy of user input parsing and validation. + +## 1.9.1 ## + +### Features + +* Added the ability to compress data when exporting it to S3-compatible storage (see the `--compression` option of the [ydb export s3](reference/ydb-cli/export-import/export-s3.md) command). +* Added the ability to manage new {{ ydb-short-name }} CLI version availability auto checks (see the `--disable-checks` and `--enable-checks` options of the [ydb version](reference/ydb-cli/version.md) command).