Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(tianmu): merge to Stonedb 5.7 stable #1919

Merged
merged 127 commits into from
Jun 30, 2023
Merged
Changes from 1 commit
Commits
Show all changes
127 commits
Select commit Hold shift + click to select a range
0a2f9be
docs(downlaod):update the docs of download(#1453)
Nliver Mar 20, 2023
dd8247b
fix(mtr):Resolve the nightly run error problem(#1458)
konghaiya Mar 21, 2023
e06793e
fix(workflow): fix can not run lcov in workflow
RingsC Mar 16, 2023
4193cf6
feat(tianmu):New configuration parameters: "tianmu_mandatory" and "ti…
konghaiya Mar 22, 2023
3dcc3fd
feat(tianmu):Discard "MANDATORY_TIANMU" and "NO_KEY_ERROR" in (sql_mode)
konghaiya Mar 22, 2023
ef58f6c
feat(tianmu):Resolve the issue of warning messages when setting new p…
konghaiya Mar 22, 2023
05db04d
fix(tianmu):(Primary/Secondary)Error 1032 occasionally occurs during …
konghaiya Mar 22, 2023
1fa5661
test(mtr): add integer/unsigned/in subquery/create temporary testcase…
davidshiz Mar 23, 2023
fd653e1
docs(v1.0.3): update the docs for v1.0.3
Nliver Mar 30, 2023
37beb82
fix(dcoker): FixDocker deployment commands(#1499)
Nliver Mar 30, 2023
45874be
feat: add Baidu statistics script(#1498)
Nliver Mar 30, 2023
d29544b
fix(website): fix Baidu statistics script(#1502)
Nliver Mar 30, 2023
9c87c73
test(mtr): Optimize parallel scheduling to execute more innodb engine…
davidshiz Mar 30, 2023
4bd09ac
fix crash when the aggregated element was decimal (#1402)
adofsauron Mar 31, 2023
ee41be0
build(deps): bump nth-check and unist-util-select in /website
dependabot[bot] Mar 31, 2023
b90030f
docs(deploy): Update the document and fix the link
Nliver Mar 31, 2023
3ac5796
update the copyright
Nliver Mar 31, 2023
e7de186
update the readme(#1324)
Nliver Mar 31, 2023
a4c5f7c
Create FUNDING.yml
stoneatomadmin Apr 3, 2023
7484c58
Update FUNDING.yml
stoneatomadmin Apr 3, 2023
e7bbe88
remove dup lines in git workflow
zzzz-vincent Apr 5, 2023
82b249a
feat(tianmu): mv func from public to protected(#1501)
zzzz-vincent Apr 4, 2023
7b03c28
style change
zzzz-vincent Apr 4, 2023
e049e3b
strip spaces
zzzz-vincent Apr 4, 2023
1675216
remove dup lines
zzzz-vincent Apr 4, 2023
59e9174
change to uint
zzzz-vincent Apr 4, 2023
d49df15
fix(tianmu):Resolve DDL and insert inot select Possible space inflati…
konghaiya Mar 20, 2023
1de9a91
feat(tianmu):Reconstruct direct insert into parallel execution, impro…
konghaiya Mar 28, 2023
bcc42ad
feat(tianmu):Add code comments for easy understanding
konghaiya Mar 28, 2023
9d45726
fix(tianmu):The myloader cann't work if the autocommit is closed. (#1…
konghaiya Apr 4, 2023
03f5df2
fix(core): fix bug: The instance occasionally crashes if both fields …
wisehead Apr 4, 2023
65e7c25
fix: page hover font style
Agility6 Apr 7, 2023
166f87a
feat(mtr): To fix the mtr usage
RingsC Apr 10, 2023
56cd1f3
fix(tianmu):The mysqld is crashed when you are starting replication.(…
konghaiya Apr 4, 2023
44d6ff8
feat(tianmu):Add delta layer mtr
konghaiya Apr 11, 2023
758bce8
bug 1538:The instance occasionally crashes when the parallel degree i…
wisehead Apr 10, 2023
8146e9b
feat(tianmu): To support vocalno framwork
RingsC Apr 10, 2023
16d2c01
docs(quickstart): add stonedb-8.0 compiling guide #1449
Double0101 Apr 3, 2023
8a4b65b
fix(website): fix website error #1449
Double0101 Apr 11, 2023
d0665f9
feat(tianmu): support volcano framewrok (#1546)
RingsC Apr 12, 2023
8e197ae
fix(tianmu): revert code, mv ret value from try block back to catch b…
Dysprosium0626 Apr 13, 2023
45e75ee
feat(tiamnu): hard code in defs.h (#1481)
zzzz-vincent Apr 6, 2023
f8cf62b
docs:update the compile guides #1562
Apr 14, 2023
09f0e40
test(mtr): add more innodb testcases and tianmu range testcase(#1196)
davidshiz Apr 16, 2023
99c5e80
fix(tianmu):Remove excess log printing and add some code comments(#1545)
konghaiya Apr 11, 2023
4b7c64b
fix(tianmu): fix mysqld crash when exec query with AggregateRough, as…
adofsauron Apr 19, 2023
e510ad3
fix(website): fix the download link of 5.7(#1518)
Nliver Apr 19, 2023
f79002e
feat(website): update the latest content(#1587)
Nliver Apr 19, 2023
6e56cef
feat(tianmu): support volcano framework (#1554)
RingsC Apr 19, 2023
875be6f
fix: max-width navbar search style
Agility6 Apr 20, 2023
c8c37fd
feat(website): upgrade the docusaurus version(#1604)
Nliver Apr 20, 2023
6b56366
fix(website): fix Roadmap module location(#1597)
Nliver Apr 20, 2023
2f6ae10
website(community): update the content
Nliver Apr 20, 2023
ba8cead
feat(website): update the logo of XinChuang(#1590)
Nliver Apr 20, 2023
0b1c15a
fix(tianmu):The instance occasionally crashes when the memory leak. (…
konghaiya Apr 17, 2023
feb4b53
fix(tianmu):Modify merge_ The assignment method of ID, delaying the a…
konghaiya Apr 18, 2023
6703037
fix(tianmu):Fix bug in delta layer initialization
konghaiya Apr 18, 2023
2e4dbdc
fix(tianmu):Resolve the issue of assertion failure caused by memory a…
konghaiya Apr 19, 2023
31919be
fix(tianmu):Code format adjustment
konghaiya Apr 20, 2023
561f653
fix(website): fix the wrong QR code(#1624)
Nliver Apr 26, 2023
29000c9
feat(tianmu):Add delta layer information output and table name output
konghaiya Apr 26, 2023
32cb03c
fix(tianmu):Perfect atomic operations for delta_table
konghaiya Apr 26, 2023
cc8a96b
feat(tianmu):Optimize delta layer merge operations to remove useless …
konghaiya Apr 26, 2023
f1a7259
fix(tianmu):assert failed on ptr == buff.get() + data_.sum_len at pac…
konghaiya Apr 26, 2023
4a41514
fix(tianmu):assert failed on oldv <= dpn_->max_i at pack_int.cpp:337 …
konghaiya Apr 26, 2023
70f15f1
feat(tianmu):Increase assertion printing information and optimize cod…
konghaiya Apr 26, 2023
587d276
fix(tianmu): fix mysqld crash when query where JOIN::propagate_de…
adofsauron Apr 27, 2023
4a35d96
fix(tianmu): fix MySQL server has gone away when exec query (#164…
adofsauron Apr 28, 2023
57e3497
fix(tianmu):Support insert ignore syntax (#1637)
konghaiya Apr 28, 2023
1ba5663
fix(tianmu): fix query input variables wrong result (#1647)
adofsauron May 5, 2023
70bdb91
fix(tianmu): fix result of the query using the subquery derived t…
adofsauron May 6, 2023
b728610
fix(tianmu): fix results of two queries using a derived table and…
adofsauron May 6, 2023
c95d263
feat(tianmu): Test cases that supplement custom variables (#1703)
adofsauron May 6, 2023
2daac7c
fix(tianmu): fix mysqld crash when assigning return values using …
adofsauron May 8, 2023
74b5b05
fix(tianmu): Insert ignore can insert duplicate values.(#1699)
konghaiya May 8, 2023
a33ba87
fix(tianmu): fix error occurred in the union all query result (#1…
adofsauron May 8, 2023
6ed8fb1
remove unused code block
zzzz-vincent May 1, 2023
8d438e1
fix bug and change test case exptected result
zzzz-vincent May 7, 2023
b16bfaa
add stonedb-8.0 compiling guide for CentOS 7.x
Xinqiangxu May 9, 2023
c4d2d2f
docs(quickstart): add stonedb-8.0 compiling guide(Chinese) for CentOS…
Xinqiangxu May 10, 2023
304df8e
fix(tianmu):Even if a primary key is defined, duplicate data may be i…
konghaiya May 10, 2023
2c96b13
add delete/drop into tianmu log stat
duanjr May 8, 2023
d9b4126
open log for all cmds
duanjr May 9, 2023
141918d
fit format
duanjr May 10, 2023
c3931fe
fix(tianmu): fix Error result set of the IN subquery with semi jo…
adofsauron May 11, 2023
65585be
doc(develop-guide): modify method for complie stonedb using docker
Dysprosium0626 May 10, 2023
20abfc7
docs:add docker compile guide of stonedb8.0.(#1780)
May 16, 2023
9b50843
feature: remove DBUG_OFF and repalce DEBUG_ASSERT with assert
duanjr May 8, 2023
72a160c
automatically formatting
duanjr May 9, 2023
9805e07
fix: fix storage of DT type
duanjr May 9, 2023
6c11356
fix incorrect result of TIME type by distinguishing the processing of…
duanjr May 11, 2023
69a49bd
fix(tinmu): fix tianmu crash when set varchar to num when order by
May 19, 2023
d99c9d5
docs(website): update the documentation for Compile StoneDB 8.0 in Do…
davidshiz May 25, 2023
2579bd4
fix(tianmu): fix up the incompatible type
RingsC May 25, 2023
0b853cd
fix(tianmu): Fix up the unknown exception after instance killed rando…
RingsC May 31, 2023
e28183b
fix(tianmu): fix up the incorrect meta-info leads unexpected behavior…
RingsC May 31, 2023
988cf1d
fix(workflow): nightly build failed #1830
hustjieke May 29, 2023
6cba6d7
feat(tianmu): revert assert() --> debug_assert() #1551
hustjieke May 31, 2023
1f43e44
feat(tianmu): fixup the default delimeter for load data (#1843)
RingsC May 31, 2023
12fa952
fix(tianmu): revert PR #1841. (#1850)
RingsC Jun 3, 2023
1b51907
fix(tianmu): fix up mtr test case for delim of load data command (#1854)
RingsC Jun 5, 2023
bde212a
fix(tianmu): fix up the `group_concat` function in tianmu (#1852)
RingsC Jun 6, 2023
1a7fe0f
fix(tianmu): To fixup the instance crashed if the result of aggregate…
RingsC Jun 6, 2023
9404338
docs(developer-guide): update the compiling guide of stonedb 8.0 for …
Xinqiangxu May 26, 2023
6b369e5
fix(tianmu): Fixup the mem leakage of aggregation function
RingsC Jun 8, 2023
f180323
fix(tianmu): fix UNION of non-matching columns (column no 0)
adofsauron Jun 14, 2023
798ca7a
test(tianmu): add order by sentence in the mtr case various_join.test
Xinqiangxu Jun 15, 2023
288262f
test(mtr): add more test cases for tianmu(#1196)
davidshiz Jun 15, 2023
c990b5f
test(mtr): add order by sentence in the mtr case various_join.test
Xinqiangxu Jun 16, 2023
9939611
ci(codecov): update the config
Nliver Jun 20, 2023
d7584f1
fix(tianmu): To suuport ignore option for update statement
RingsC Jun 14, 2023
5606c87
ci(codecov): update the codecov congfig
Nliver Jun 20, 2023
105b730
docs(intro): update the support for 8.0
Nliver Jun 20, 2023
186f694
wokflow(codecov): Filter out excess code files
Nliver Jun 25, 2023
7d03695
workflow(coverage): Update the lcov running logic
Nliver Jun 26, 2023
73dcfec
fix(tianmu): default value of the field take unaffect in load #1865
Double0101 Jun 19, 2023
a4600c4
fix(tianmu): To support union(all) the statement which is without fro…
RingsC Jun 15, 2023
6a560a0
fix(tianmu): To remove unnessary optimization in tianmu
RingsC Jun 27, 2023
60a8cf7
fix(tianmu): hotfix corruption in ValueOrNull under multi-thread
RingsC Jun 9, 2023
6349898
fix(tianmu): incorrect result when using where expr and args > bigint…
hustjieke Jun 9, 2023
34b73a1
fix(tianmu): add TIME_to_ulonglong_time_round process and fix up prec…
Xinqiangxu Jun 19, 2023
deb0600
fix(tianmu): fix format using clang-format #792
hustjieke Oct 24, 2022
25b767b
feat: rm files after rebase leftover #1217
hustjieke Jan 14, 2023
422bd73
Merge branch 'stonedb-5.7-stable' of github.com:RingsC/stonedb into HEAD
RingsC Jun 29, 2023
969435a
Merge branch 'stoneatom:stonedb-5.7-stable' into stonedb-5.7-stable
RingsC Jun 29, 2023
69213ff
fix(sql,tianmu):fix when binlog format is row, the load data statemen…
Jun 25, 2023
4494273
Merge branch 'stonedb-5.7-dev' into stable-binlog
RingsC Jun 30, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix(tianmu): hotfix corruption in ValueOrNull under multi-thread
In multi-thread aggregation, ExpressionColumn will occur double free due to without
protection. Thread A will do ValueOrNull::operator ==, but in thread B, it will try to
free it. Therefore, it leads to instance crash.
RingsC authored and mergify[bot] committed Jun 28, 2023

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
commit 60a8cf71f3632cba2bd28ab1f6f94101abece140
14 changes: 14 additions & 0 deletions mysql-test/suite/tianmu/r/issue1855.result
Original file line number Diff line number Diff line change
@@ -10,3 +10,17 @@ SUM(LENGTH(p_val))
SELECT SUM(LENGTH(p_val)/2) FROM ttt;
SUM(LENGTH(p_val)/2)
3.0000
Variable_name Value
tianmu_groupby_parallel_degree 0
tianmu_groupby_parallel_rows_minimum 655360
Variable_name Value
tianmu_groupby_parallel_degree 4
tianmu_groupby_parallel_rows_minimum 100
SUM(LENGTH(p_id))
158949
SUM(LENGTH(p_id)/2)
79474.5000
SUM(LENGTH(p_val))
38901
SUM(LENGTH(p_val)/2)
19450.5000
50 changes: 49 additions & 1 deletion mysql-test/suite/tianmu/t/issue1855.test
Original file line number Diff line number Diff line change
@@ -32,8 +32,56 @@ SELECT SUM(LENGTH(p_val)) FROM ttt;
SELECT SUM(LENGTH(p_val)/2) FROM ttt;

--disable_query_log
DROP TABLE ttt;

--DELIMITER //
CREATE PROCEDURE insert_data()
wholeblock:BEGIN
DECLARE x INT;
DECLARE str VARCHAR(50);
DECLARE str1 VARCHAR(50);
SET x = 0;
SET str = 'this is ';
SET str1 = '';

loop_label: LOOP
IF x > 10000 THEN
LEAVE loop_label;
END IF;

SET str1 = substring(md5(rand()), 1, 10);
SET str = CONCAT(str,str1,',');
SET str = CONCAT(str,x,',');

INSERT INTO ttt VALUES (str, x);
SET str ='';
SET x = x + 1;
ITERATE loop_label;
END LOOP;
END//

--DELIMITER ;

CALL insert_data();

DROP PROCEDURE insert_data;

SHOW VARIABLES LIKE "%tianmu_groupby_parallel%";

SET GLOBAL tianmu_groupby_parallel_rows_minimum = 100;
SET GLOBAL tianmu_groupby_parallel_degree = 4;

SHOW VARIABLES LIKE "%tianmu_groupby_parallel%";

SELECT SUM(LENGTH(p_id)) FROM ttt;

SELECT SUM(LENGTH(p_id)/2) FROM ttt;

SELECT SUM(LENGTH(p_val)) FROM ttt;

SELECT SUM(LENGTH(p_val)/2) FROM ttt;


DROP TABLE ttt;
DROP DATABASE issue1855_test_db;
--enable_query_log

13 changes: 6 additions & 7 deletions storage/tianmu/core/value_or_null.cpp
Original file line number Diff line number Diff line change
@@ -23,6 +23,7 @@

namespace Tianmu {
namespace core {

void ValueOrNull::SetBString(const types::BString &tianmu_s) {
Clear();
if (!tianmu_s.IsNull()) {
@@ -44,7 +45,7 @@ void ValueOrNull::SetBString(const types::BString &tianmu_s) {
}

void ValueOrNull::MakeStringOwner() {
if (!sp || string_owner)
if (!sp || !len || string_owner)
return;

char *tmp = new (std::nothrow) char[len + 1];
@@ -70,17 +71,14 @@ void ValueOrNull::GetBString(types::BString &tianmu_s) const {
tianmu_s = rcs_null;
} else {
// copy either from sp or x
if (sp)
tianmu_s = types::BString(sp, len, true);
else
tianmu_s = types::TianmuNum(x).ToBString();
tianmu_s = (sp) ? types::BString(sp, len, true) : types::TianmuNum(x).ToBString();
tianmu_s.MakePersistent();
}
}

ValueOrNull::ValueOrNull(ValueOrNull const &von)
: x(von.x), len(von.len), string_owner(von.string_owner), null(von.null) {
if (string_owner) {
if (string_owner && von.sp && len > 0) {
sp = new (std::nothrow) char[len + 1];

if (sp) {
@@ -99,7 +97,7 @@ ValueOrNull &ValueOrNull::operator=(ValueOrNull const &von) {
if (this == &von)
return *this;

if (von.string_owner) {
if (von.string_owner && von.sp) {
sp = new (std::nothrow) char[von.len + 1];
if (sp) {
std::memset(sp, '\0', von.len + 1);
@@ -140,5 +138,6 @@ void ValueOrNull::Swap(ValueOrNull &von) {
std::swap(string_owner, von.string_owner);
}
}

} // namespace core
} // namespace Tianmu
1 change: 1 addition & 0 deletions storage/tianmu/core/value_or_null.h
Original file line number Diff line number Diff line change
@@ -102,6 +102,7 @@ class ValueOrNull final {

string_owner = false;
null = true;
len = 0;
}
}

2 changes: 1 addition & 1 deletion storage/tianmu/handler/ha_tianmu.cpp
Original file line number Diff line number Diff line change
@@ -2623,7 +2623,7 @@ static MYSQL_SYSVAR_BOOL(groupby_speedup, tianmu_sysvar_groupby_speedup, PLUGIN_
static MYSQL_SYSVAR_UINT(groupby_parallel_degree, tianmu_sysvar_groupby_parallel_degree, PLUGIN_VAR_INT,
"group by parallel degree, number of worker threads", nullptr, nullptr, 8, 0, INT32_MAX, 0);
static MYSQL_SYSVAR_ULONGLONG(groupby_parallel_rows_minimum, tianmu_sysvar_groupby_parallel_rows_minimum,
PLUGIN_VAR_LONGLONG, "group by parallel minimum rows", nullptr, nullptr, 655360, 655360,
PLUGIN_VAR_LONGLONG, "group by parallel minimum rows", nullptr, nullptr, 655360, 100,
INT64_MAX, 0);
static MYSQL_SYSVAR_UINT(slow_query_record_interval, tianmu_sysvar_slow_query_record_interval, PLUGIN_VAR_INT,
"slow Query Threshold of recording tianmu logs, in seconds", nullptr, nullptr, 0, 0, INT32_MAX,
1 change: 1 addition & 0 deletions storage/tianmu/optimizer/aggregation_algorithm.cpp
Original file line number Diff line number Diff line change
@@ -1072,6 +1072,7 @@ void AggregationWorkerEnt::DistributeAggreTaskAverage(MIIterator &mit, uint64_t
for (uint i = 0; i < vTask.size(); ++i) {
if (dims.NoDimsUsed() == 0)
dims.SetAll();

auto &mii = taskIterator.emplace_back(mit, true);
mii.SetTaskNum(vTask.size());
mii.SetTaskId(i);
14 changes: 11 additions & 3 deletions storage/tianmu/optimizer/aggregator_basic.cpp
Original file line number Diff line number Diff line change
@@ -24,15 +24,23 @@
namespace Tianmu {
namespace core {
void AggregatorSum64::PutAggregatedValue(unsigned char *buf, int64_t v, int64_t factor) {
std::scoped_lock scp_lk(aggr_mtx);

stats_updated = false;
int64_t *p = (int64_t *)buf;
if (*p == common::NULL_VALUE_64) {
*p = 0;
}
double overflow_check = double(*p) + double(v) * factor;

long double overflow_check = double(*p) + double(v) * factor;
if (overflow_check > std::numeric_limits<std::streamsize>::max() ||
overflow_check < std::numeric_limits<std::streamsize>::min())
throw common::NotImplementedException("Aggregation overflow.");
overflow_check < std::numeric_limits<std::streamsize>::min()) {
char str_buff[1024] = {'\0'};
sprintf(str_buff, "Aggregation overflow for long double. over_check: %Lf", overflow_check);

throw common::InternalException(str_buff);
}

*p += v * factor;
}

2 changes: 2 additions & 0 deletions storage/tianmu/optimizer/aggregator_basic.h
Original file line number Diff line number Diff line change
@@ -18,6 +18,7 @@
#define TIANMU_CORE_AGGREGATOR_BASIC_H_
#pragma once

#include <mutex>
#include "optimizer/aggregator.h"

namespace Tianmu {
@@ -87,6 +88,7 @@ class AggregatorSum64 : public TIANMUAggregator {
}

private:
std::mutex aggr_mtx;
int64_t pack_sum;
int64_t pack_min; // min and max are used to check whether a pack may update
// sum (i.e. both 0 means "no change")
2 changes: 2 additions & 0 deletions storage/tianmu/optimizer/group_table.cpp
Original file line number Diff line number Diff line change
@@ -569,9 +569,11 @@ bool GroupTable::PutAggregatedValue(int col, int64_t row, MIIterator &mit, int64
DEBUG_ASSERT(gdistinct[col]);
if (vc[col]->IsNull(mit))
return true; // omit nulls

GDTResult res = gdistinct[col]->Add(row, mit);
if (res == GDTResult::GDT_EXISTS)
return true; // value found, do not aggregate it again

if (res == GDTResult::GDT_FULL) {
// if (gdistinct[col]->AlreadyFull())
// not_full = false; // disable also the main grouping table (if it is a
1 change: 1 addition & 0 deletions storage/tianmu/vc/column_share.h
Original file line number Diff line number Diff line change
@@ -140,6 +140,7 @@ class ColumnShare final {
bool has_filter_hist = false;
bool has_filter_bloom = false;
};

} // namespace core
} // namespace Tianmu

5 changes: 5 additions & 0 deletions storage/tianmu/vc/expr_column.cpp
Original file line number Diff line number Diff line change
@@ -16,6 +16,7 @@
*/

#include "expr_column.h"
#include <mutex>
#include "core/mysql_expression.h"
#include "optimizer/compile/compiled_query.h"
#include "vc/tianmu_attr.h"
@@ -131,6 +132,9 @@ bool ExpressionColumn::FeedArguments(const core::MIIterator &mit) {
}

int64_t ExpressionColumn::GetValueInt64Impl(const core::MIIterator &mit) {
static std::mutex scp_mutex;
std::scoped_lock lock(scp_mutex);

if (FeedArguments(mit))
last_val_ = expr_->Evaluate();

@@ -152,6 +156,7 @@ int64_t ExpressionColumn::GetValueInt64Impl(const core::MIIterator &mit) {
val_it.second->Clear_SP();
}
}

return last_val_->Get64();
}