Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Main: Table metadata in RocksDB #12

Closed
wants to merge 183 commits into from
Closed
Changes from 1 commit
Commits
Show all changes
183 commits
Select commit Hold shift + click to select a range
16b423c
Debug logs
arpang Sep 23, 2022
9d590e7
WIP: Prototype
arpang Sep 26, 2022
6ca6663
Compilation fix
arpang Sep 26, 2022
51f4c0b
Startup issue fix
arpang Sep 26, 2022
fcf35a0
Merge remote-tracking branch 'origin/master' into arpan/tablet-metadata
arpang Sep 26, 2022
44b381c
Temp fix to bypass system tables
arpang Sep 28, 2022
09904de
Master table schemas writing to DocDB
arpang Sep 28, 2022
b8217af
Writes metadata table to super block, helper function to read table info
arpang Sep 30, 2022
a620bea
Metadata table id field in super block
arpang Oct 1, 2022
9daf6e9
Map clear bug fix
arpang Oct 1, 2022
a61404c
Basic working end to end
arpang Oct 1, 2022
bdc51c9
WIP: Skips writing tableinfo to superblock
arpang Oct 3, 2022
c36f674
[#14221][WIP][Prototype] DocDB: Decouple table schema from tablet met…
arpang Oct 3, 2022
f08635f
Missing cotable id bug fix
arpang Oct 4, 2022
01b217d
Restore colocation id to 1
arpang Oct 4, 2022
7439321
Prototype working end to end
arpang Oct 4, 2022
e6de02e
Log cleanup
arpang Oct 4, 2022
daf657c
Log cleanup
arpang Oct 4, 2022
dd8d9d4
Revert unnecessary auto format changes
arpang Oct 4, 2022
1aaabee
Merge remote-tracking branch 'origin/master' into arpan/tablet-metadata
arpang Oct 4, 2022
948063a
Release compilation fix
arpang Oct 24, 2022
6fb78d8
Jitter optimization
arpang Oct 28, 2022
c1074f2
Upgrade path for master and tserver
arpang Nov 4, 2022
81d5ec1
Jitter and backoff controlling gflags
arpang Nov 8, 2022
4775290
Lint error fix
arpang Nov 8, 2022
b94a83b
Reserved prefix, initial_primary_table and fixed upgrade path
arpang Nov 15, 2022
573c111
Lint erros fix
arpang Nov 15, 2022
a350c95
WIP: Reserved prefix fixes
arpang Nov 17, 2022
e3c5855
Rserved prefix: startup issue fixed
arpang Nov 17, 2022
e98082f
Reserved prefix: Table not found fix
arpang Nov 17, 2022
d2d237f
Fixes unexpected range component size
arpang Nov 17, 2022
257624f
Merge remote-tracking branch 'origin/master' into arpan/tablet-metadata
arpang Nov 18, 2022
221d5ec
Fixes lint issues
arpang Nov 18, 2022
bca8138
Handle remove table
arpang Nov 18, 2022
775b221
Fixes the seg fault issue
arpang Nov 21, 2022
7fce33f
Documentation
arpang Nov 22, 2022
75ef429
Lint issue fix
arpang Nov 22, 2022
63dee2b
None code changes
arpang Nov 22, 2022
b384245
Metadata change
arpang Nov 22, 2022
6063ac6
Minor changes
arpang Nov 22, 2022
4b115e9
DDL operation working
arpang Nov 22, 2022
f4f86d7
Minor refactoring
arpang Nov 23, 2022
032a74b
Bug fix: invalid serialized string while doc operation apply
arpang Nov 24, 2022
53bfabe
Minor changes
arpang Nov 24, 2022
2791cb5
Alter wal retention support
arpang Nov 24, 2022
6be86c3
ChangeMetadataOperation replay
arpang Nov 26, 2022
6cc7de3
Lint fixes
arpang Nov 26, 2022
bfdd3fc
Table split support
arpang Nov 30, 2022
ce36cc7
Seg fault fix
arpang Nov 30, 2022
aca4bc5
Compaction packed row bug fix
arpang Dec 2, 2022
5d90a75
Updated todo
arpang Dec 2, 2022
a3be3a1
Merge remote-tracking branch 'origin/master' into arpan/tablet-metadata
arpang Dec 5, 2022
17293d4
Lint issues fixed
arpang Dec 5, 2022
7bb8ae9
Cleanup
arpang Dec 8, 2022
5a996c6
Rename docdb to rocksdb
arpang Dec 8, 2022
b10a411
Rename back primary_table_info to table_info
arpang Dec 8, 2022
0619097
Undoes changes in sys catalog
arpang Dec 8, 2022
227e603
Minor change
arpang Dec 8, 2022
6587006
Refactoring
arpang Dec 8, 2022
d9a62f3
Pragma once
arpang Dec 8, 2022
70c4146
Minor changes
arpang Dec 8, 2022
f2b67e7
Fixes read restart error
arpang Dec 15, 2022
9d262bf
Minor todo
arpang Dec 15, 2022
4e79b65
Merge remote-tracking branch 'origin/master' into arpan/tablet-metadata
arpang Dec 15, 2022
7dec3ac
Minor change
arpang Dec 15, 2022
35d4ce5
Autoflag kLocalPersisted
arpang Dec 15, 2022
06be5ce
Minor changes
arpang Dec 15, 2022
e91632b
Minor changes
arpang Dec 15, 2022
7d5ed6e
Merge remote-tracking branch 'origin/master' into arpan/tablet-metadata
arpang Dec 16, 2022
62916c6
Build fix
arpang Dec 16, 2022
9182f49
Handles OldSchemaGC
arpang Dec 16, 2022
eb3f43d
Handles setting namespace_id
arpang Dec 16, 2022
84efc4c
Removes code duplication
arpang Dec 16, 2022
cf0ece5
Fix
arpang Dec 19, 2022
18cc448
Merge remote-tracking branch 'origin/master' into arpan/tablet-metadata
arpang Dec 19, 2022
c2c9942
Build fix
arpang Dec 19, 2022
8096944
Phabricator build failure: debugging attempt
arpang Dec 19, 2022
f3153f9
Introduced is_metadata_schema in Schema class
arpang Dec 20, 2022
2e6b315
Builds and persists metadata schema
arpang Dec 21, 2022
dad1fa3
Removes unnecessary includes
arpang Dec 21, 2022
31f7e27
Bug fix
arpang Dec 21, 2022
49baab4
Handles metadata key in ToString()
arpang Dec 21, 2022
6b62d28
Changes column types from string to binary
arpang Dec 21, 2022
e3bc5d7
Compaction bug fix
arpang Dec 21, 2022
b3d74fb
Undoes unnecessary changes
arpang Dec 21, 2022
ec7d8d6
Undoes unnecessary changes
arpang Dec 21, 2022
1af0363
Undoes unnecessary changes
arpang Dec 21, 2022
6178324
Fixes schema tests
arpang Dec 21, 2022
29f63eb
Partially fixes remote bootstrap issue
arpang Dec 21, 2022
390343c
Fixes packed row tests
arpang Dec 21, 2022
374dddf
Fixes packed row tests
arpang Dec 21, 2022
21bc035
Scan bug fix
arpang Dec 22, 2022
383cd94
Fixes Key parsing failed error
arpang Dec 24, 2022
8aab318
Merge remote-tracking branch 'origin/master' into arpan/tablet-metadata
arpang Dec 24, 2022
d70d335
Adds a todo
arpang Dec 24, 2022
d2c6828
Fixes PgLibPqTest.ReplayDeletedTableInColocatedDB
arpang Dec 24, 2022
73a2fc6
Reserve vector size
arpang Dec 26, 2022
6dc8475
Fixes stack-use-after-scope error
arpang Dec 26, 2022
e288c43
Fixes Check failed: hybrid_time_.is_valid() error
arpang Dec 26, 2022
1df1fda
Updates todos
arpang Dec 26, 2022
8c0d268
Minor updates
arpang Dec 27, 2022
13e0fad
Handles minor todos
arpang Dec 27, 2022
1ffc20e
Updates todos
arpang Dec 27, 2022
765227c
Removes unnecessary logs
arpang Dec 27, 2022
cf4c982
Minor fixes
arpang Dec 27, 2022
a7d3a04
Cleanup and add todos
arpang Dec 27, 2022
29f4676
Self review: WIP
arpang Dec 27, 2022
bdc77b1
WIP: Self review
arpang Dec 27, 2022
1cefab5
Self review
arpang Dec 27, 2022
4e8fa06
Style fixes
arpang Dec 27, 2022
93f6b56
Fixes read time and takes lock before writing
arpang Dec 28, 2022
1a38a2d
Fixes logic to detect ts tablets
arpang Dec 28, 2022
1a2328d
Updates todos
arpang Dec 28, 2022
32c4f20
Sets correct isolation level
arpang Dec 28, 2022
eacf284
Removes unnecessary changes
arpang Dec 28, 2022
b4ccddb
Removes unnecessary changes
arpang Dec 28, 2022
42e7973
Removes unnecessary changes
arpang Dec 28, 2022
52ebc75
Calls AddFollowerPending and Replicated before and after replaying Ch…
arpang Dec 29, 2022
65b69e0
Dedupes code
arpang Dec 29, 2022
9c55c41
Style fixes
arpang Dec 29, 2022
f6cd694
Updates DocKey comments
arpang Dec 29, 2022
6720683
Minor change
arpang Dec 29, 2022
cbd5c68
Makes the metadata key range column
arpang Jan 1, 2023
325c0da
Merge remote-tracking branch 'origin/master' into arpan/tablet-metadata
arpang Jan 1, 2023
a5e5fab
Moves SysRowEntryType to common namespace
arpang Jan 2, 2023
1e0e985
Moves SysRowEntryType to common namespace
arpang Jan 2, 2023
c1a47c2
Moves SysRowEntryType to common namespace
arpang Jan 2, 2023
2c43c6e
Read entries corresponding to TABLET_TABLE only when loading metadata
arpang Jan 2, 2023
776a446
Self review fixes
arpang Jan 2, 2023
009a7bf
Fixes Infinite loop detected error
arpang Jan 3, 2023
43874ed
Fixes infinite loop detected
arpang Jan 4, 2023
edbdc82
Removes unnecessary comments
arpang Jan 4, 2023
e907194
Addresses some review comments
arpang Jan 5, 2023
007684b
Addresses some review comments
arpang Jan 5, 2023
356e0a2
Addresses some review comments
arpang Jan 5, 2023
3629373
Sets the flag to false
arpang Jan 9, 2023
ec95cd9
Replay request: checks if metadata is in rocksdb
arpang Jan 9, 2023
f37929b
Merge remote-tracking branch 'origin/master' into arpan/tablet-metadata
arpang Jan 9, 2023
4d24e40
Revert "Moves SysRowEntryType to common namespace"
arpang Jan 18, 2023
f77fdb4
SysRowEntryType in both common and master
arpang Jan 18, 2023
0508761
Updates todos
arpang Jan 18, 2023
42161dc
Minor changes
arpang Jan 18, 2023
c015d1e
Minor changes
arpang Jan 18, 2023
9a7a557
Updates todos
arpang Jan 18, 2023
18a9cdf
Replaces bool with empty type
arpang Jan 18, 2023
0258868
Fixes failing TestWalRetentionSet
arpang Jan 19, 2023
cc20dd4
Merge remote-tracking branch 'origin/master' into arpan/tablet-metadata
arpang Jan 23, 2023
7523ebc
Minor fix
arpang Jan 23, 2023
c66ac0b
ChangeMetadataDocOperation takes TableInfoPtr parameter
arpang Jan 23, 2023
38ac252
TableInfoPtr as ChangeMetadataDocOperation class member
arpang Jan 23, 2023
328419d
Bug fix: delete the entire doc key instead of just metadata column
arpang Jan 23, 2023
35eaf64
Updates the todos
arpang Jan 23, 2023
f124184
Minor error message fix
arpang Jan 23, 2023
5d60467
Address minor review comment
arpang Jan 24, 2023
073516a
Temporarily setting the flag to true
arpang Jan 24, 2023
4a37b9a
Sets the default flag to false
arpang Jan 24, 2023
7ee0148
Removes used field
arpang Jan 25, 2023
ee6b467
WIP: Decoupling bootstrap and primary table fields
arpang Jan 25, 2023
22986d0
Merge remote-tracking branch 'origin/master' into arpan/tablet-metadata
arpang Jan 26, 2023
4890025
Fixes lint issues
arpang Jan 26, 2023
77e0ad2
Merge branch 'arpan/tablet-metadata' into arpan/primary-table
arpang Jan 26, 2023
35b30eb
Primary table in rocksdb: basic funtionality working
arpang Jan 26, 2023
4f8b8ba
Prototype: Primary table metadata in rocksdb
arpang Jan 26, 2023
26f7fb0
Reply bug fixed: shouldn't check for table's existence
arpang Jan 26, 2023
341ccf9
Moves metadata load instead tablet bootstrap
arpang Jan 27, 2023
be751a0
Minor fixes
arpang Jan 27, 2023
98113d4
Undoes log class changes
arpang Jan 27, 2023
c52597c
Undoes unnecessary changes
arpang Jan 27, 2023
e7a1957
Merge remote-tracking branch 'origin/master' into arpan/tablet-metadata
arpang Jan 27, 2023
94c08d5
Merge branch 'arpan/tablet-metadata' into arpan/primary-table
arpang Jan 27, 2023
205d1c1
Handles remote bootstrap
arpang Jan 27, 2023
5bcee65
Null pointer bug fix
arpang Jan 28, 2023
5cbead9
Skip schema version in heartbeat till table metadata is loaded
arpang Jan 28, 2023
73cda9d
Minor bug fix
arpang Jan 28, 2023
8752172
Set table & namespace name attribute if already available
arpang Jan 28, 2023
895ac3a
Sets table and namespace atts to tablet metrics entity
arpang Jan 28, 2023
83cd792
Fixes packed row tests
arpang Jan 28, 2023
8121c8c
Fixes check failed has_primary_tablet_type()
arpang Jan 29, 2023
fd7e8b1
Adds a todo
arpang Jan 30, 2023
ae24c14
Adds a todo for future
arpang Jan 30, 2023
4c51748
Cleanup
arpang Feb 1, 2023
10d1048
Revert "Revert "Moves SysRowEntryType to common namespace""
arpang Feb 1, 2023
98c8d1a
Minor changes
arpang Feb 1, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Jitter and backoff controlling gflags
arpang committed Nov 8, 2022
commit 81d5ec1412d595403660dc76a172317af8374a6c
1 change: 1 addition & 0 deletions src/yb/util/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -135,6 +135,7 @@ set(UTIL_SRCS
async_task_util.cc
atomic.cc
background_task.cc
backoff_waiter.cc
bitmap.cc
bitmap.cc
bloom_filter.cc
46 changes: 46 additions & 0 deletions src/yb/util/backoff_waiter.cc
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
// Copyright (c) YugaByte, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except
// in compliance with the License. You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software distributed under the License
// is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
// or implied. See the License for the specific language governing permissions and limitations
// under the License.

#include "yb/util/backoff_waiter.h"
#include "yb/util/flag_tags.h"

#include <algorithm>
#include <string>

#include <glog/logging.h>

using std::string;

DEFINE_test_flag(int32, max_jitter, 0, "Max backoff jitter");
DEFINE_test_flag(
int32, backoff_start_exponent, 0, "Initial exponent of 2 that backoff starts with");

namespace yb {

template <class Clock>
typename Clock::duration GenericBackoffWaiter<Clock>::DelayForTime(TimePoint now) const {
Duration max_wait = std::min(deadline_ - now, max_wait_);
// 1st retry delayed 2^4 of base delays, 2nd 2^5 base delays, etc..
Duration attempt_delay =
base_delay_ * (attempt_ >= 29 ? std::numeric_limits<int32_t>::max()
: 1LL << (attempt_ + FLAGS_TEST_backoff_start_exponent));
Duration jitter = std::chrono::milliseconds(RandomUniformInt(0, FLAGS_TEST_max_jitter));

LOG(INFO) << "Total delay: " << (attempt_delay + jitter) << " attempt_delay: " << attempt_delay
<< " jitter: " << jitter;
return std::min(attempt_delay + jitter, max_wait);
}

template class GenericBackoffWaiter<std::chrono::steady_clock>;
template class GenericBackoffWaiter<CoarseMonoClock>;

} // namespace yb
22 changes: 12 additions & 10 deletions src/yb/util/backoff_waiter.h
Original file line number Diff line number Diff line change
@@ -66,16 +66,18 @@ class GenericBackoffWaiter {
return DelayForTime(Clock::now());
}

Duration DelayForTime(TimePoint now) const {
Duration max_wait = std::min(deadline_ - now, max_wait_);
// 1st retry delayed 2^4 of base delays, 2nd 2^5 base delays, etc..
Duration attempt_delay =
base_delay_ * (attempt_ >= 29 ? std::numeric_limits<int32_t>::max() : 1LL << (attempt_));
// Duration jitter = std::chrono::milliseconds(RandomUniformInt(0, 10));
LOG(INFO) << "Delay: " << (attempt_delay) << " attempt_delay: " << attempt_delay;
// << " jitter: " << jitter;
return std::min(attempt_delay, max_wait);
}
Duration DelayForTime(TimePoint now) const;

// Duration DelayForTime(TimePoint now) const {
// Duration max_wait = std::min(deadline_ - now, max_wait_);
// // 1st retry delayed 2^4 of base delays, 2nd 2^5 base delays, etc..
// Duration attempt_delay =
// base_delay_ * (attempt_ >= 29 ? std::numeric_limits<int32_t>::max() : 1LL << (attempt_));
// // Duration jitter = std::chrono::milliseconds(RandomUniformInt(0, 10));
// LOG(INFO) << "Delay: " << (attempt_delay) << " attempt_delay: " << attempt_delay;
// // << " jitter: " << jitter;
// return std::min(attempt_delay, max_wait);
// }

size_t attempt() const {
return attempt_;