fix stonedb for optimize exists subquery and deep memory copy #369 (#330, #329, #340, #332)

[adofsauron]

Summary about this PR

• Code submission Repository: https://github.com/adofsauron/stonedb-dev-20220805
• Code consolidation repository: https://github.com/stoneatom/stonedb
• branch: stonedb-5.7
• related issuse:
• bug: slow-SQL-Q4-when subquery use exists sql on 10GB data spend 40 mininus and happen OOM on 100GB data #330
• bug: slow-SQL-Q2-when subquery use like sql on 10GB data and happen OOM on 100GB data #329
• question: SQL num 18 use too much RAM because of deep copy on parall hash join, is deep copy really necessary? #340
• bug: slow-SQL-Q17-when subquery use avg sql spend too much time #332

mtr测试结果:

[100%] shutdown_report                          [ pass ]       
--------------------------------------------------------------------------
The servers were restarted 22 times
Spent 4.815 of 218 seconds executing testcases

Check of testcase failed for: tianmu.group_concat2 tianmu.issue475 tianmu.issue469 tianmu.issue405 tianmu.truncate tianmu.issue502 tianmu.issue429 tianmu.concat tianmu.issue468 tianmu.casewhen tianmu.insert tianmu.lz4 tianmu.add_column tianmu.group_concat tianmu.orquery tianmu.issue430 tianmu.orquery1 tianmu.issue431 tianmu.issue464 tianmu.issue496 tianmu.issue467 tianmu.drop_column tianmu.issue417

Completed: All 24 tests were successful.

exists子句测试结果

/stonedb57/install/bin/mysql -D tpch -e 'select
                            o_orderpriority,
                            count(*) as order_count
                        from
                            orders
                        where
                            o_orderdate >= date '\''1993-07-01'\''
                            and o_orderdate < date '\''1993-07-01'\'' + interval '\''3'\'' month
                            and exists (
                                select
                                    *
                                from
                                    lineitem
                                where
                                    l_orderkey = o_orderkey
                                    and l_commitdate < l_receiptdate
                            )
                        group by
                            o_orderpriority
                        order by
                            o_orderpriority ;'
Warning:
PID/TID switch overriding SYSTEM
o_orderpriority	order_count
1-URGENT	114839
2-HIGH	114276
3-MEDIUM	114716
4-NOT SPECIFIED	114913
5-LOW	114927

Q2子句测试

Issue Number: close #330 #329

Tests Check List

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Changelog

• New Feature
• Bug Fix
• Improvement
• Performance Improvement
• Build/Testing/CI/CD
• Documentation
• Not for changelog (changelog entry is not required)

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features

[mergify]

This pull request's title should follow requirements next. @adofsauron please check it 👇.

Valid format:

fix(vc): fix sth..... (#3306)
  ^         ^---------^  ^----^
  |         |            |
  |         +            +-> you issue id.
  |         |
  |         +-> Summary in present tense.
  |
  +-------> Type: feat, fix, docs, workflow, style, refactor, test, website, chore

Valid types:

• feat: new feature for stonedb
• fix: bug fix for stonedb
• docs: changes to the documentation
• workflow: ci/cd in .github
• refactor: refactoring production code, eg. renaming a variable
• style: formatting, missing semi colons, etc; no production code change
• test: adding missing tests, refactoring tests; no production code change
• website
• chore: updating grunt tasks etc; no production code change

[mergify]

Thanks for the contribution!
I have applied any labels matching special text in your PR Changelog.

Please review the labels and make any necessary changes.

[isredstar]

LGTM

[adofsauron]

摘要:
记录在并行hash join时候去除DimensionGroupMultiMaterialized的深拷贝

核心函数:

DimensionGroupMultiMaterialized::Clone

DimensionGroup *DimensionGroupMultiMaterialized::Clone(bool shallow) {
  DimensionGroupMultiMaterialized *new_value = new DimensionGroupMultiMaterialized(no_obj, dims_used_, power_);
  if (shallow) return new_value;
  for (int index = 0; index < dims_count_; ++index) {
    MultiIndexTable *tables = dim_tables_[index];
    if (tables) {
      tables->Lock();
      new_value->dim_tables_[index] = new MultiIndexTable(*tables);
      tables->Unlock();
    }
  }
  return new_value;
}

优化后:

DimensionGroupMultiMaterialized::Clone

DimensionGroup *DimensionGroupMultiMaterialized::Clone(bool shallow) {
  DimensionGroupMultiMaterialized *new_value = new DimensionGroupMultiMaterialized(no_obj, dims_used_, power_, shallow);
  for (int index = 0; index < dims_count_; ++index) {
    MultiIndexTable *tables = dim_tables_[index];
    if (tables) {
      tables->Lock();
      if (shallow) {
        new_value->dim_tables_[index] = tables;
      } else {
        new_value->dim_tables_[index] = new MultiIndexTable(*tables);
      }

      tables->Unlock();
    }
  }
  return new_value;
}

DimensionGroupMultiMaterialized的析构函数

DimensionGroupMultiMaterialized::~DimensionGroupMultiMaterialized() {
  if (is_shallow_memory) {
    return;
  }

  for (auto it : dim_tables_) delete it;
}

执行测试:

结论:

1. 执行结果无异常
2. 执行耗时Q18， 由6min削减为4min30s
3. 内存稳定保持在60GB (执行前后都是)

[andyli029]

Fix the commit log.