Add export checksums #12728

pixcc · 2024-12-18T13:18:41Z

Changelog category

Not for changelog (changelog entry is not required)

ydb/core/tx/datashard/export_checksum.h

github-actions · 2024-12-24T05:07:46Z

⚪ 2024-12-24 05:07:45 UTC Pre-commit check linux-x86_64-release-asan for 271593a has started.
⚪ 2024-12-24 05:07:56 UTC Artifacts will be uploaded here
⚫ 2024-12-24 05:08:28 UTC Check cancelled

github-actions · 2024-12-24T05:07:49Z

⚪ 2024-12-24 05:07:48 UTC Pre-commit check linux-x86_64-relwithdebinfo for 271593a has started.
⚪ 2024-12-24 05:07:59 UTC Artifacts will be uploaded here
⚫ 2024-12-24 05:08:28 UTC Check cancelled

github-actions · 2024-12-24T05:09:54Z

⚪ 2024-12-24 05:09:53 UTC Pre-commit check linux-x86_64-release-asan for ece4e97 has started.
⚪ 2024-12-24 05:09:57 UTC Artifacts will be uploaded here
🔴 2024-12-24 05:11:47 UTC Graph compare failed, see the logs.

github-actions · 2024-12-24T05:10:36Z

⚪ 2024-12-24 05:10:35 UTC Pre-commit check linux-x86_64-relwithdebinfo for ece4e97 has started.
⚪ 2024-12-24 05:10:47 UTC Artifacts will be uploaded here
🔴 2024-12-24 05:12:42 UTC Graph compare failed, see the logs.

github-actions · 2024-12-24T06:00:04Z

⚪ 2024-12-24 06:00:03 UTC Pre-commit check linux-x86_64-release-asan for 3f44021 has started.
⚪ 2024-12-24 06:00:16 UTC Artifacts will be uploaded here
🔴 2024-12-24 06:02:08 UTC Graph compare failed, see the logs.

github-actions · 2024-12-24T06:00:04Z

⚪ 2024-12-24 06:00:04 UTC Pre-commit check linux-x86_64-relwithdebinfo for 3f44021 has started.
⚪ 2024-12-24 06:00:16 UTC Artifacts will be uploaded here
🔴 2024-12-24 06:02:12 UTC Graph compare failed, see the logs.

github-actions · 2024-12-24T06:25:42Z

⚪ 2024-12-24 06:25:41 UTC Pre-commit check linux-x86_64-relwithdebinfo for 8140e3d has started.
⚪ 2024-12-24 06:25:52 UTC Artifacts will be uploaded here
⚪ 2024-12-24 06:29:06 UTC ya make is running...
🟡 2024-12-24 07:56:57 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
31022	28023	0	11	2862	126

⚪ 2024-12-24 07:59:30 UTC ya make is running... (failed tests rerun, try 2)
🟡 2024-12-24 08:11:31 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
219 (only retried tests)	103	0	5	0	111

⚪ 2024-12-24 08:11:40 UTC ya make is running... (failed tests rerun, try 3)
🔴 2024-12-24 08:23:11 UTC Some tests failed, follow the links below.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
162 (only retried tests)	47	0	5	0	110

🟢 2024-12-24 08:23:20 UTC Build successful.
🟡 2024-12-24 08:23:42 UTC ydbd size 2.1 GiB changed* by +112.6 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `18ed18e`	merge: `8140e3d`	diff	diff %
ydbd size	2 235 030 744 Bytes	2 235 146 032 Bytes	+112.6 KiB	+0.005%
ydbd stripped size	477 983 088 Bytes	478 003 120 Bytes	+19.6 KiB	+0.004%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-12-24T06:25:52Z

⚪ 2024-12-24 06:25:51 UTC Pre-commit check linux-x86_64-release-asan for 8140e3d has started.
⚪ 2024-12-24 06:26:02 UTC Artifacts will be uploaded here
⚪ 2024-12-24 06:29:19 UTC ya make is running...
🟡 2024-12-24 08:12:02 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
14050	13962	0	25	15	48

🟢 2024-12-24 08:13:13 UTC Build successful.
🟡 2024-12-24 08:13:39 UTC ydbd size 3.6 GiB changed* by +166.5 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `18ed18e`	merge: `8140e3d`	diff	diff %
ydbd size	3 865 652 864 Bytes	3 865 823 336 Bytes	+166.5 KiB	+0.004%
ydbd stripped size	1 350 678 768 Bytes	1 350 732 848 Bytes	+52.8 KiB	+0.004%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-12-24T08:59:48Z

⚪ 2024-12-24 08:59:47 UTC Pre-commit check linux-x86_64-relwithdebinfo for a8f14ec has started.
⚪ 2024-12-24 08:59:53 UTC Artifacts will be uploaded here
⚪ 2024-12-24 09:03:12 UTC ya make is running...
🟡 2024-12-24 10:29:19 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
31019	28068	0	3	2831	117

⚪ 2024-12-24 10:31:54 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-12-24 10:44:00 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
167 (only retried tests)	55	0	0	0	112

🟢 2024-12-24 10:44:07 UTC Build successful.
🟡 2024-12-24 10:44:24 UTC ydbd size 2.1 GiB changed* by +112.6 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `7cd35f3`	merge: `a8f14ec`	diff	diff %
ydbd size	2 235 028 664 Bytes	2 235 143 936 Bytes	+112.6 KiB	+0.005%
ydbd stripped size	477 983 088 Bytes	478 003 120 Bytes	+19.6 KiB	+0.004%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-12-24T09:03:09Z

⚪ 2024-12-24 09:03:08 UTC Pre-commit check linux-x86_64-release-asan for a8f14ec has started.
⚪ 2024-12-24 09:03:20 UTC Artifacts will be uploaded here
⚪ 2024-12-24 09:06:33 UTC ya make is running...
🟡 2024-12-24 10:41:52 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
14050	13961	0	28	12	49

🟡 2024-12-24 10:42:31 UTC ydbd size 3.6 GiB changed* by +166.4 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `7cd35f3`	merge: `a8f14ec`	diff	diff %
ydbd size	3 865 652 520 Bytes	3 865 822 920 Bytes	+166.4 KiB	+0.004%
ydbd stripped size	1 350 679 408 Bytes	1 350 733 616 Bytes	+52.9 KiB	+0.004%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

ydb/core/tx/datashard/backup_restore_traits.h

ydb/core/tx/datashard/export_checksum.h

github-actions · 2024-12-24T14:10:57Z

⚪ 2024-12-24 14:10:56 UTC Pre-commit check linux-x86_64-relwithdebinfo for c3e057a has started.
⚪ 2024-12-24 14:11:09 UTC Artifacts will be uploaded here
⚪ 2024-12-24 14:14:25 UTC ya make is running...
🔴 2024-12-24 14:16:19 UTC Build failed, see the logs. Also see fail summary

github-actions · 2024-12-24T14:11:10Z

⚪ 2024-12-24 14:11:10 UTC Pre-commit check linux-x86_64-release-asan for c3e057a has started.
⚪ 2024-12-24 14:11:20 UTC Artifacts will be uploaded here
⚪ 2024-12-24 14:14:36 UTC ya make is running...
🔴 2024-12-24 14:16:02 UTC Build failed, see the logs. Also see fail summary

github-actions · 2024-12-24T14:18:42Z

⚪ 2024-12-24 14:18:41 UTC Pre-commit check linux-x86_64-release-asan for aa312e2 has started.
⚪ 2024-12-24 14:19:16 UTC Artifacts will be uploaded here
⚪ 2024-12-24 14:23:07 UTC ya make is running...
🟡 2024-12-24 15:52:57 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
14061	13988	0	18	6	49

🟢 2024-12-24 15:54:21 UTC Build successful.
🟡 2024-12-24 15:54:47 UTC ydbd size 3.6 GiB changed* by +170.4 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `e432d9c`	merge: `aa312e2`	diff	diff %
ydbd size	3 866 232 704 Bytes	3 866 407 216 Bytes	+170.4 KiB	+0.005%
ydbd stripped size	1 350 957 856 Bytes	1 351 016 160 Bytes	+56.9 KiB	+0.004%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-12-24T14:18:48Z

⚪ 2024-12-24 14:18:47 UTC Pre-commit check linux-x86_64-relwithdebinfo for aa312e2 has started.
⚪ 2024-12-24 14:19:34 UTC Artifacts will be uploaded here
⚪ 2024-12-24 14:23:09 UTC ya make is running...
🟡 2024-12-24 15:48:52 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
31037	28088	0	3	2831	115

⚪ 2024-12-24 15:51:43 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-12-24 16:03:36 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
168 (only retried tests)	56	0	0	1	111

🟢 2024-12-24 16:03:46 UTC Build successful.
🟡 2024-12-24 16:04:04 UTC ydbd size 2.1 GiB changed* by +117.6 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `e432d9c`	merge: `aa312e2`	diff	diff %
ydbd size	2 235 317 224 Bytes	2 235 437 632 Bytes	+117.6 KiB	+0.005%
ydbd stripped size	478 064 016 Bytes	478 088 208 Bytes	+23.6 KiB	+0.005%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

ydb/core/tx/datashard/export_s3_uploader.cpp

ydb/core/tx/datashard/export_s3_buffer_raw.h

github-actions · 2024-12-25T11:07:54Z

⚪ 2024-12-25 11:07:54 UTC Pre-commit check linux-x86_64-relwithdebinfo for e8a0e9d has started.
⚪ 2024-12-25 11:08:05 UTC Artifacts will be uploaded here
⚪ 2024-12-25 11:11:17 UTC ya make is running...
⚫ 2024-12-25 11:44:44 UTC Check cancelled

github-actions · 2024-12-25T11:08:01Z

⚪ 2024-12-25 11:08:01 UTC Pre-commit check linux-x86_64-release-asan for e8a0e9d has started.
⚪ 2024-12-25 11:08:13 UTC Artifacts will be uploaded here
⚪ 2024-12-25 11:11:25 UTC ya make is running...
⚫ 2024-12-25 11:44:44 UTC Check cancelled

ydb/core/tx/datashard/export_s3_uploader.cpp

CyberROFL · 2024-12-25T11:38:56Z

ydb/core/tx/datashard/export_s3_uploader.cpp

+            << ": self# " << this->SelfId()
+            << ", result# " << result);
+
+        if (!CheckResult(result, TStringBuf("PutObject (checksum)"))) {


Обрати внимание, что в случае ошибки при загрузке контрольной суммы, загрузка будет перезапущена. 1) Я бы воспроизвел это в тесте и убедился, что после рестарта все работает корректно; 2) оптимизировал это место, чтобы поретраить только загрузку контрольной суммы, а не всех данных.

Всё это можно отдельным pr.

github-actions · 2024-12-25T11:46:07Z

⚪ 2024-12-25 11:46:07 UTC Pre-commit check linux-x86_64-relwithdebinfo for 730325a has started.
⚪ 2024-12-25 11:46:15 UTC Artifacts will be uploaded here
⚪ 2024-12-25 11:49:27 UTC ya make is running...
🟡 2024-12-25 12:51:47 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
31050	28092	0	2	2835	121

⚪ 2024-12-25 12:54:32 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-12-25 13:06:22 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
180 (only retried tests)	62	0	0	1	117

🟢 2024-12-25 13:06:29 UTC Build successful.
🟡 2024-12-25 13:06:50 UTC ydbd size 2.1 GiB changed* by +108.3 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `fd6bcad`	merge: `730325a`	diff	diff %
ydbd size	2 225 799 264 Bytes	2 225 910 120 Bytes	+108.3 KiB	+0.005%
ydbd stripped size	467 334 160 Bytes	467 349 648 Bytes	+15.1 KiB	+0.003%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-12-25T11:46:11Z

⚪ 2024-12-25 11:46:11 UTC Pre-commit check linux-x86_64-release-asan for 730325a has started.
⚪ 2024-12-25 11:46:29 UTC Artifacts will be uploaded here
⚪ 2024-12-25 11:49:41 UTC ya make is running...
🟡 2024-12-25 12:57:27 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
14069	13988	0	19	7	55

🟢 2024-12-25 12:58:39 UTC Build successful.
🟡 2024-12-25 12:59:02 UTC ydbd size 3.6 GiB changed* by +170.4 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `fd6bcad`	merge: `730325a`	diff	diff %
ydbd size	3 864 178 544 Bytes	3 864 353 056 Bytes	+170.4 KiB	+0.005%
ydbd stripped size	1 347 268 464 Bytes	1 347 326 064 Bytes	+56.2 KiB	+0.004%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

pixcc requested a review from CyberROFL December 18, 2024 13:18

pixcc self-assigned this Dec 18, 2024

github-actions bot added the not-for-changelog label Dec 18, 2024

CyberROFL reviewed Dec 19, 2024

View reviewed changes

ydb/core/tx/datashard/export_checksum.h Outdated Show resolved Hide resolved

ydb/core/tx/datashard/export_checksum.h Outdated Show resolved Hide resolved

pixcc force-pushed the export-checksums branch from 7ab172c to 9dc18f0 Compare December 24, 2024 05:04

pixcc force-pushed the export-checksums branch from 83f427d to c4794f7 Compare December 24, 2024 05:56

CyberROFL reviewed Dec 24, 2024

View reviewed changes

ydb/core/tx/datashard/backup_restore_traits.h Outdated Show resolved Hide resolved

ydb/core/tx/datashard/export_checksum.h Outdated Show resolved Hide resolved

ydb/core/tx/datashard/export_checksum.h Outdated Show resolved Hide resolved

CyberROFL reviewed Dec 24, 2024

View reviewed changes

pixcc added 8 commits December 25, 2024 11:05

Add export checksums

70cbc40

Add checksum interface

63a89aa

Init metadata version

8cfc5eb

Remove peerdir

ffaaed8

Fix canonization

2577313

Fix multipart

592910d

Rename file to suffix

9ab3139

Fix export_checksum

0db204b

pixcc added 2 commits December 25, 2024 11:05

Fix build

829a393

Fix after review

7a09062

pixcc force-pushed the export-checksums branch from b96825e to 7a09062 Compare December 25, 2024 11:05

CyberROFL previously approved these changes Dec 25, 2024

View reviewed changes

Fix after review

30cd329

pixcc dismissed CyberROFL’s stale review via 30cd329 December 25, 2024 11:44

CyberROFL approved these changes Dec 25, 2024

View reviewed changes

pixcc merged commit 2edc9b5 into ydb-platform:main Dec 25, 2024
12 checks passed

GrigoriyPA mentioned this pull request Dec 27, 2024

YQ-3949 fix pq source sensors names #13088

Merged

GrigoriyPA mentioned this pull request Jan 5, 2025

KqpRun improved multi query mode and storage settings #13177

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add export checksums #12728

Add export checksums #12728

pixcc commented Dec 18, 2024

github-actions bot commented Dec 24, 2024 •

edited

Loading

github-actions bot commented Dec 24, 2024 •

edited

Loading

github-actions bot commented Dec 24, 2024 •

edited

Loading

github-actions bot commented Dec 24, 2024 •

edited

Loading

github-actions bot commented Dec 24, 2024 •

edited

Loading

github-actions bot commented Dec 24, 2024 •

edited

Loading

github-actions bot commented Dec 24, 2024 •

edited

Loading

github-actions bot commented Dec 24, 2024 •

edited

Loading

github-actions bot commented Dec 24, 2024 •

edited

Loading

github-actions bot commented Dec 24, 2024 •

edited

Loading

github-actions bot commented Dec 24, 2024 •

edited

Loading

github-actions bot commented Dec 24, 2024 •

edited

Loading

github-actions bot commented Dec 24, 2024 •

edited

Loading

github-actions bot commented Dec 24, 2024 •

edited

Loading

github-actions bot commented Dec 25, 2024 •

edited

Loading

github-actions bot commented Dec 25, 2024 •

edited

Loading

CyberROFL Dec 25, 2024

github-actions bot commented Dec 25, 2024 •

edited

Loading

github-actions bot commented Dec 25, 2024 •

edited

Loading

Add export checksums #12728

Add export checksums #12728

Conversation

pixcc commented Dec 18, 2024

Changelog category

github-actions bot commented Dec 24, 2024 • edited Loading

github-actions bot commented Dec 24, 2024 • edited Loading

github-actions bot commented Dec 24, 2024 • edited Loading

github-actions bot commented Dec 24, 2024 • edited Loading

github-actions bot commented Dec 24, 2024 • edited Loading

github-actions bot commented Dec 24, 2024 • edited Loading

github-actions bot commented Dec 24, 2024 • edited Loading

github-actions bot commented Dec 24, 2024 • edited Loading

github-actions bot commented Dec 24, 2024 • edited Loading

github-actions bot commented Dec 24, 2024 • edited Loading

github-actions bot commented Dec 24, 2024 • edited Loading

github-actions bot commented Dec 24, 2024 • edited Loading

github-actions bot commented Dec 24, 2024 • edited Loading

github-actions bot commented Dec 24, 2024 • edited Loading

github-actions bot commented Dec 25, 2024 • edited Loading

github-actions bot commented Dec 25, 2024 • edited Loading

CyberROFL Dec 25, 2024

Choose a reason for hiding this comment

github-actions bot commented Dec 25, 2024 • edited Loading

github-actions bot commented Dec 25, 2024 • edited Loading

github-actions bot commented Dec 24, 2024 •

edited

Loading

github-actions bot commented Dec 24, 2024 •

edited

Loading

github-actions bot commented Dec 24, 2024 •

edited

Loading

github-actions bot commented Dec 24, 2024 •

edited

Loading

github-actions bot commented Dec 24, 2024 •

edited

Loading

github-actions bot commented Dec 24, 2024 •

edited

Loading

github-actions bot commented Dec 24, 2024 •

edited

Loading

github-actions bot commented Dec 24, 2024 •

edited

Loading

github-actions bot commented Dec 24, 2024 •

edited

Loading

github-actions bot commented Dec 24, 2024 •

edited

Loading

github-actions bot commented Dec 24, 2024 •

edited

Loading

github-actions bot commented Dec 24, 2024 •

edited

Loading

github-actions bot commented Dec 24, 2024 •

edited

Loading

github-actions bot commented Dec 24, 2024 •

edited

Loading

github-actions bot commented Dec 25, 2024 •

edited

Loading

github-actions bot commented Dec 25, 2024 •

edited

Loading

github-actions bot commented Dec 25, 2024 •

edited

Loading

github-actions bot commented Dec 25, 2024 •

edited

Loading