Fix pagestorage v3 ddl problem. #4691

jiaqizho · 2022-04-18T07:14:42Z

Signed-off-by: jiaqizho [email protected]

What problem does this PR solve?

Issue Number: ref #3594 close #4712

Problem Summary:

Run ddl test with enable_ps_v3 will get random nullptr error.
Also we still have some uninit errors when I run valgrind.

==24236== Syscall param pwrite64(buf) points to uninitialised byte(s)
==24236==    at 0xE0D8FC3: ??? (syscall-template.S:81)
==24236==    by 0x47D2EEE: DB::PosixWritableFile::pwrite(char*, unsigned long, long) const (PosixWritableFile.cpp:91)
==24236==    by 0x4B4107F: void DB::PageUtil::writeFile<std::shared_ptr<DB::WritableFile> >(std::shared_ptr<DB::WritableFile>&, unsigned long, char*, unsigned long, std::shared_ptr<DB::WriteLimiter> const&, bool) (PageUtil.h:178)
==24236==    by 0x4BB3A04: DB::PS::V3::LogWriter::flush(std::shared_ptr<DB::WriteLimiter> const&) (LogWriter.cpp:72)
==24236==    by 0x4BB3C68: DB::PS::V3::LogWriter::addRecord(DB::ReadBuffer&, unsigned long, std::shared_ptr<DB::WriteLimiter> const&) (LogWriter.cpp:143)
==24236==    by 0x4EFEF0E: DB::PS::V3::WALStore::apply(DB::PS::V3::PageEntriesEdit const&, std::shared_ptr<DB::WriteLimiter> const&) (WALStore.cpp:92)
==24236==    by 0x4EFEFF7: DB::PS::V3::WALStore::apply(DB::PS::V3::PageEntriesEdit&, DB::PS::V3::PageVersionType const&, std::shared_ptr<DB::WriteLimiter> const&) (WALStore.cpp:72)
==24236==    by 0x4EEF736: DB::PS::V3::PageDirectory::apply(DB::PS::V3::PageEntriesEdit&&, std::shared_ptr<DB::WriteLimiter> const&) (PageDirectory.cpp:964)
==24236==    by 0x4BADC49: DB::PS::V3::PageStorageImpl::writeImpl(DB::WriteBatch&&, std::shared_ptr<DB::WriteLimiter> const&) (PageStorageImpl.cpp:96)
==24236==    by 0x49A8AE1: DB::PageStorage::write(DB::WriteBatch&&, std::shared_ptr<DB::WriteLimiter> const&) (PageStorage.h:184)
==24236==    by 0x49B55DE: DB::DM::WriteBatches::rollbackWrittenLogAndData() (WriteBatches.h:154)
==24236==    by 0x49B58DA: DB::DM::WriteBatches::~WriteBatches() (WriteBatches.h:81)
==24236==  Address 0x2b2997af is 31 bytes inside a block of size 32,768 alloc'd
==24236==    at 0xA986F73: malloc (vg_replace_malloc.c:309)
==24236==    by 0x4B0528: Allocator<false>::alloc(unsigned long, unsigned long) (Allocator.cpp:94)
==24236==    by 0x4BB38C9: DB::PS::V3::LogWriter::LogWriter(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::shared_ptr<DB::FileProvider> const&, unsigned int, bool, bool) (LogWriter.cpp:48)
==24236==    by 0x4F008D4: std::_MakeUniq<DB::PS::V3::LogWriter>::__single_object std::make_unique<DB::PS::V3::LogWriter, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::shared_ptr<DB::FileProvider>&, unsigned int const&, bool, bool&>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::shared_ptr<DB::FileProvider>&, unsigned int const&, bool&&, bool&) (unique_ptr.h:825)
==24236==    by 0x4EFED15: DB::PS::V3::WALStore::createLogWriter(std::pair<unsigned int, unsigned int> const&, bool) (WALStore.cpp:134)
==24236==    by 0x4EFEEC3: DB::PS::V3::WALStore::apply(DB::PS::V3::PageEntriesEdit const&, std::shared_ptr<DB::WriteLimiter> const&) (WALStore.cpp:87)
==24236==    by 0x4EFEFF7: DB::PS::V3::WALStore::apply(DB::PS::V3::PageEntriesEdit&, DB::PS::V3::PageVersionType const&, std::shared_ptr<DB::WriteLimiter> const&) (WALStore.cpp:72)
==24236==    by 0x4EEF736: DB::PS::V3::PageDirectory::apply(DB::PS::V3::PageEntriesEdit&&, std::shared_ptr<DB::WriteLimiter> const&) (PageDirectory.cpp:964)
==24236==    by 0x4BADC49: DB::PS::V3::PageStorageImpl::writeImpl(DB::WriteBatch&&, std::shared_ptr<DB::WriteLimiter> const&) (PageStorageImpl.cpp:96)
==24236==    by 0x49A8AE1: DB::PageStorage::write(DB::WriteBatch&&, std::shared_ptr<DB::WriteLimiter> const&) (PageStorage.h:184)
==24236==    by 0x49B9788: DB::DM::WriteBatches::writeLogAndData() (WriteBatches.h:114)
==24236==    by 0x4E75B72: DB::DM::ColumnFileFlushTask::prepare(DB::DM::WriteBatches&) (ColumnFileFlushTask.cpp:58)
==24236== 



==24236== Use of uninitialised value of size 8
==24236==    at 0x527A6F2: crc64::_detail::update16(unsigned long, std::array<unsigned char, 16ul> const&) (crc64_table.h:1520)
==24236==    by 0x527A949: crc64::_detail::update_table(unsigned long, void const*, unsigned long) (crc64_table.h:1566)
==24236==    by 0x527A9F4: unsigned long crc64::_detail::update_fast<128ul, unsigned long (*)(unsigned long, void const*, unsigned long)>(unsigned long (*)(unsigned long, void const*, unsigned long), unsigned long, void const*, unsigned long) (crc64_fast.h:61)
==24236==    by 0x527AA25: crc64::Digest::Digest(crc64::Mode)::{lambda(unsigned long, void const*, unsigned long)#1}::operator()(unsigned long, void const*, unsigned long) const (crc64.cpp:54)
==24236==    by 0x527AA3E: crc64::Digest::Digest(crc64::Mode)::{lambda(unsigned long, void const*, unsigned long)#1}::_FUN(unsigned long, void const*, unsigned long) (crc64.cpp:54)
==24236==    by 0x47E06F1: crc64::Digest::update(void const*, unsigned long) (crc64.h:33)
==24236==    by 0x47E07BD: DB::Digest::CRC64::update(void const*, unsigned long) (Checksum.h:104)
==24236==    by 0x4BB36B9: DB::PS::V3::LogWriter::emitPhysicalRecord(DB::PS::V3::Format::RecordType, DB::ReadBuffer&, unsigned long) (LogWriter.cpp:191)
==24236==    by 0x4BB3B82: DB::PS::V3::LogWriter::addRecord(DB::ReadBuffer&, unsigned long, std::shared_ptr<DB::WriteLimiter> const&) (LogWriter.cpp:135)
==24236==    by 0x4EFEF0E: DB::PS::V3::WALStore::apply(DB::PS::V3::PageEntriesEdit const&, std::shared_ptr<DB::WriteLimiter> const&) (WALStore.cpp:92)
==24236==    by 0x4EFEFF7: DB::PS::V3::WALStore::apply(DB::PS::V3::PageEntriesEdit&, DB::PS::V3::PageVersionType const&, std::shared_ptr<DB::WriteLimiter> const&) (WALStore.cpp:72)
==24236==    by 0x4EEF736: DB::PS::V3::PageDirectory::apply(DB::PS::V3::PageEntriesEdit&&, std::shared_ptr<DB::WriteLimiter> const&) (PageDirectory.cpp:964)
==24236==

root cause:

==8154== 
==8154== Thread 30 BkgPool6:
==8154== Invalid write of size 1
==8154==    at 0x10970FBA: memset (vg_replace_strmem.c:1270)
==8154==    by 0x446C07C: void DB::PODArrayBase<1ul, 4096ul, Allocator<false>, 15ul, 16ul>::alloc<>(unsigned long) (in /data1/lidezhu/tiflash-scripts/integrated/nodes/8284/tiflash1/tiflash/tiflash)
==8154==    by 0x446BF6B: void DB::PODArrayBase<1ul, 4096ul, Allocator<false>, 15ul, 16ul>::realloc<>(unsigned long) (in /data1/lidezhu/tiflash-scripts/integrated/nodes/8284/tiflash1/tiflash/tiflash)
==8154==    by 0x446BE91: void DB::PODArrayBase<1ul, 4096ul, Allocator<false>, 15ul, 16ul>::reserve<>(unsigned long) (in /data1/lidezhu/tiflash-scripts/integrated/nodes/8284/tiflash1/tiflash/tiflash)
==8154==    by 0xBD4BB66: DB::DataTypeString::deserializeBinaryBulk(DB::IColumn&, DB::ReadBuffer&, unsigned long, double) const (in /data1/lidezhu/tiflash-scripts/integrated/nodes/8284/tiflash1/tiflash/tiflash)
==8154==    by 0x46A0B97: DB::IDataType::deserializeBinaryBulkWithMultipleStreams(DB::IColumn&, std::__1::function<DB::ReadBuffer* (std::__1::vector<DB::IDataType::Substream, std::__1::allocator<DB::IDataType::Substream> > const&)> const&, unsigned long, double, bool, std::__1::vector<DB::IDataType::Substream, std::__1::allocator<DB::IDataType::Substream> >&) const (in /data1/lidezhu/tiflash-scripts/integrated/nodes/8284/tiflash1/tiflash/tiflash)
==8154==    by 0xBD3A4BA: DB::DataTypeNullable::deserializeBinaryBulkWithMultipleStreams(DB::IColumn&, std::__1::function<DB::ReadBuffer* (std::__1::vector<DB::IDataType::Substream, std::__1::allocator<DB::IDataType::Substream> > const&)> const&, unsigned long, double, bool, std::__1::vector<DB::IDataType::Substream, std::__1::allocator<DB::IDataType::Substream> >&) const (in /data1/lidezhu/tiflash-scripts/integrated/nodes/8284/tiflash1/tiflash/tiflash)
==8154==    by 0xBCD1643: DB::IDataType::deserializeBinaryBulkWithMultipleStreams(DB::IColumn&, std::__1::function<DB::ReadBuffer* (std::__1::vector<DB::IDataType::Substream, std::__1::allocator<DB::IDataType::Substream> > const&)> const&, unsigned long, double, bool, std::__1::vector<DB::IDataType::Substream, std::__1::allocator<DB::IDataType::Substream> >&&) const (in /data1/lidezhu/tiflash-scripts/integrated/nodes/8284/tiflash1/tiflash/tiflash)
==8154==    by 0xD708A01: DB::DM::deserializeColumn(DB::IColumn&, std::__1::shared_ptr<DB::IDataType const> const&, DB::ByteBuffer const&, unsigned long) (in /data1/lidezhu/tiflash-scripts/integrated/nodes/8284/tiflash1/tiflash/tiflash)
==8154==    by 0xD6FDC70: DB::DM::ColumnFileTiny::readFromDisk(DB::PageReader const&, std::__1::vector<DB::DM::ColumnDefine, std::__1::allocator<DB::DM::ColumnDefine> > const&, unsigned long, unsigned long) const (in /data1/lidezhu/tiflash-scripts/integrated/nodes/8284/tiflash1/tiflash/tiflash)
==8154==    by 0xD6FDEB8: DB::DM::ColumnFileTiny::fillColumns(DB::PageReader const&, std::__1::vector<DB::DM::ColumnDefine, std::__1::allocator<DB::DM::ColumnDefine> > const&, unsigned long, std::__1::vector<COWPtr<DB::IColumn>::immutable_ptr<DB::IColumn>, std::__1::allocator<COWPtr<DB::IColumn>::immutable_ptr<DB::IColumn> > >&) const (in /data1/lidezhu/tiflash-scripts/integrated/nodes/8284/tiflash1/tiflash/tiflash)
==8154==    by 0xD6FF132: DB::DM::ColumnFileTinyReader::readRows(std::__1::vector<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn>, std::__1::allocator<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn> > >&, unsigned long, unsigned long, DB::DM::RowKeyRange const*) (in /data1/lidezhu/tiflash-scripts/integrated/nodes/8284/tiflash1/tiflash/tiflash)
==8154==  Address 0x192df26f is 15 bytes after a block of size 0 alloc'd
==8154==    at 0x10966EE5: malloc (vg_replace_malloc.c:380)
==8154==    by 0x4472B4A: Allocator<false>::alloc(unsigned long, unsigned long) (in /data1/lidezhu/tiflash-scripts/integrated/nodes/8284/tiflash1/tiflash/tiflash)
==8154==    by 0x446C044: void DB::PODArrayBase<1ul, 4096ul, Allocator<false>, 15ul, 16ul>::alloc<>(unsigned long) (in /data1/lidezhu/tiflash-scripts/integrated/nodes/8284/tiflash1/tiflash/tiflash)
==8154==    by 0x446BF6B: void DB::PODArrayBase<1ul, 4096ul, Allocator<false>, 15ul, 16ul>::realloc<>(unsigned long) (in /data1/lidezhu/tiflash-scripts/integrated/nodes/8284/tiflash1/tiflash/tiflash)
==8154==    by 0x446BE91: void DB::PODArrayBase<1ul, 4096ul, Allocator<false>, 15ul, 16ul>::reserve<>(unsigned long) (in /data1/lidezhu/tiflash-scripts/integrated/nodes/8284/tiflash1/tiflash/tiflash)
==8154==    by 0xBD4BB66: DB::DataTypeString::deserializeBinaryBulk(DB::IColumn&, DB::ReadBuffer&, unsigned long, double) const (in /data1/lidezhu/tiflash-scripts/integrated/nodes/8284/tiflash1/tiflash/tiflash)
==8154==    by 0x46A0B97: DB::IDataType::deserializeBinaryBulkWithMultipleStreams(DB::IColumn&, std::__1::function<DB::ReadBuffer* (std::__1::vector<DB::IDataType::Substream, std::__1::allocator<DB::IDataType::Substream> > const&)> const&, unsigned long, double, bool, std::__1::vector<DB::IDataType::Substream, std::__1::allocator<DB::IDataType::Substream> >&) const (in /data1/lidezhu/tiflash-scripts/integrated/nodes/8284/tiflash1/tiflash/tiflash)
==8154==    by 0xBD3A4BA: DB::DataTypeNullable::deserializeBinaryBulkWithMultipleStreams(DB::IColumn&, std::__1::function<DB::ReadBuffer* (std::__1::vector<DB::IDataType::Substream, std::__1::allocator<DB::IDataType::Substream> > const&)> const&, unsigned long, double, bool, std::__1::vector<DB::IDataType::Substream, std::__1::allocator<DB::IDataType::Substream> >&) const (in /data1/lidezhu/tiflash-scripts/integrated/nodes/8284/tiflash1/tiflash/tiflash)
==8154==    by 0xBCD1643: DB::IDataType::deserializeBinaryBulkWithMultipleStreams(DB::IColumn&, std::__1::function<DB::ReadBuffer* (std::__1::vector<DB::IDataType::Substream, std::__1::allocator<DB::IDataType::Substream> > const&)> const&, unsigned long, double, bool, std::__1::vector<DB::IDataType::Substream, std::__1::allocator<DB::IDataType::Substream> >&&) const (in /data1/lidezhu/tiflash-scripts/integrated/nodes/8284/tiflash1/tiflash/tiflash)
==8154==    by 0xD708A01: DB::DM::deserializeColumn(DB::IColumn&, std::__1::shared_ptr<DB::IDataType const> const&, DB::ByteBuffer const&, unsigned long) (in /data1/lidezhu/tiflash-scripts/integrated/nodes/8284/tiflash1/tiflash/tiflash)
==8154==    by 0xD6FDC70: DB::DM::ColumnFileTiny::readFromDisk(DB::PageReader const&, std::__1::vector<DB::DM::ColumnDefine, std::__1::allocator<DB::DM::ColumnDefine> > const&, unsigned long, unsigned long) const (in /data1/lidezhu/tiflash-scripts/integrated/nodes/8284/tiflash1/tiflash/tiflash)
==8154==    by 0xD6FDEB8: DB::DM::ColumnFileTiny::fillColumns(DB::PageReader const&, std::__1::vector<DB::DM::ColumnDefine, std::__1::allocator<DB::DM::ColumnDefine> > const&, unsigned long, std::__1::vector<COWPtr<DB::IColumn>::immutable_ptr<DB::IColumn>, std::__1::allocator<COWPtr<DB::IColumn>::immutable_ptr<DB::IColumn> > >&) const (in /data1/lidezhu/tiflash-scripts/integrated/nodes/8284/tiflash1/tiflash/tiflash)
==8154== 
==8154== Invalid write of size 8
==8154==    at 0xBD4BE3E: void DB::deserializeBinarySSE2<4>(DB::PODArray<unsigned char, 4096ul, Allocator<false>, 15ul, 16ul>&, DB::PODArray<unsigned long, 4096ul, Allocator<false>, 15ul, 16ul>&, DB::ReadBuffer&, unsigned long) (in /data1/lidezhu/tiflash-scripts/integrated/nodes/8284/tiflash1/tiflash/tiflash)
==8154==    by 0xBD4BBB3: DB::DataTypeString::deserializeBinaryBulk(DB::IColumn&, DB::ReadBuffer&, unsigned long, double) const (in /data1/lidezhu/tiflash-scripts/integrated/nodes/8284/tiflash1/tiflash/tiflash)
==8154==    by 0x46A0B97: DB::IDataType::deserializeBinaryBulkWithMultipleStreams(DB::IColumn&, std::__1::function<DB::ReadBuffer* (std::__1::vector<DB::IDataType::Substream, std::__1::allocator<DB::IDataType::Substream> > const&)> const&, unsigned long, double, bool, std::__1::vector<DB::IDataType::Substream, std::__1::allocator<DB::IDataType::Substream> >&) const (in /data1/lidezhu/tiflash-scripts/integrated/nodes/8284/tiflash1/tiflash/tiflash)
==8154==    by 0xBD3A4BA: DB::DataTypeNullable::deserializeBinaryBulkWithMultipleStreams(DB::IColumn&, std::__1::function<DB::ReadBuffer* (std::__1::vector<DB::IDataType::Substream, std::__1::allocator<DB::IDataType::Substream> > const&)> const&, unsigned long, double, bool, std::__1::vector<DB::IDataType::Substream, std::__1::allocator<DB::IDataType::Substream> >&) const (in /data1/lidezhu/tiflash-scripts/integrated/nodes/8284/tiflash1/tiflash/tiflash)
==8154==    by 0xBCD1643: DB::IDataType::deserializeBinaryBulkWithMultipleStreams(DB::IColumn&, std::__1::function<DB::ReadBuffer* (std::__1::vector<DB::IDataType::Substream, std::__1::allocator<DB::IDataType::Substream> > const&)> const&, unsigned long, double, bool, std::__1::vector<DB::IDataType::Substream, std::__1::allocator<DB::IDataType::Substream> >&&) const (in /data1/lidezhu/tiflash-scripts/integrated/nodes/8284/tiflash1/tiflash/tiflash)
==8154==    by 0xD708A01: DB::DM::deserializeColumn(DB::IColumn&, std::__1::shared_ptr<DB::IDataType const> const&, DB::ByteBuffer const&, unsigned long) (in /data1/lidezhu/tiflash-scripts/integrated/nodes/8284/tiflash1/tiflash/tiflash)
==8154==    by 0xD6FDC70: DB::DM::ColumnFileTiny::readFromDisk(DB::PageReader const&, std::__1::vector<DB::DM::ColumnDefine, std::__1::allocator<DB::DM::ColumnDefine> > const&, unsigned long, unsigned long) const (in /data1/lidezhu/tiflash-scripts/integrated/nodes/8284/tiflash1/tiflash/tiflash)
==8154==    by 0xD6FDEB8: DB::DM::ColumnFileTiny::fillColumns(DB::PageReader const&, std::__1::vector<DB::DM::ColumnDefine, std::__1::allocator<DB::DM::ColumnDefine> > const&, unsigned long, std::__1::vector<COWPtr<DB::IColumn>::immutable_ptr<DB::IColumn>, std::__1::allocator<COWPtr<DB::IColumn>::immutable_ptr<DB::IColumn> > >&) const (in /data1/lidezhu/tiflash-scripts/integrated/nodes/8284/tiflash1/tiflash/tiflash)
==8154==    by 0xD6FF132: DB::DM::ColumnFileTinyReader::readRows(std::__1::vector<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn>, std::__1::allocator<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn> > >&, unsigned long, unsigned long, DB::DM::RowKeyRange const*) (in /data1/lidezhu/tiflash-scripts/integrated/nodes/8284/tiflash1/tiflash/tiflash)
==8154==    by 0xD6F1B84: DB::DM::ColumnFileSetReader::readRows(std::__1::vector<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn>, std::__1::allocator<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn> > >&, unsigned long, unsigned long, DB::DM::RowKeyRange const*) (in /data1/lidezhu/tiflash-scripts/integrated/nodes/8284/tiflash1/tiflash/tiflash)
==8154==    by 0xD74B3CF: DB::DM::DeltaValueReader::readRows(std::__1::vector<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn>, std::__1::allocator<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn> > >&, unsigned long, unsigned long, DB::DM::RowKeyRange const*) (in /data1/lidezhu/tiflash-scripts/integrated/nodes/8284/tiflash1/tiflash/tiflash)
==8154==    by 0xD5CA55D: DB::DM::DeltaMergeBlockInputStream<DB::DM::DeltaValueReader, DB::DM::DTCompactedEntries<55ul, 20ul, 3ul>::Iterator, false>::writeInsertFromDelta(std::__1::vector<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn>, std::__1::allocator<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn> > >&, unsigned long&) (in /data1/lidezhu/tiflash-scripts/integrated/nodes/8284/tiflash1/tiflash/tiflash)
==8154==  Address 0x192df270 is 16 bytes after a block of size 0 alloc'd
==8154==    at 0x10966EE5: malloc (vg_replace_malloc.c:380)
==8154==    by 0x4472B4A: Allocator<false>::alloc(unsigned long, unsigned long) (in /data1/lidezhu/tiflash-scripts/integrated/nodes/8284/tiflash1/tiflash/tiflash)
==8154==    by 0x446C044: void DB::PODArrayBase<1ul, 4096ul, Allocator<false>, 15ul, 16ul>::alloc<>(unsigned long) (in /data1/lidezhu/tiflash-scripts/integrated/nodes/8284/tiflash1/tiflash/tiflash)
==8154==    by 0x446BF6B: void DB::PODArrayBase<1ul, 4096ul, Allocator<false>, 15ul, 16ul>::realloc<>(unsigned long) (in /data1/lidezhu/tiflash-scripts/integrated/nodes/8284/tiflash1/tiflash/tiflash)
==8154==    by 0x446BE91: void DB::PODArrayBase<1ul, 4096ul, Allocator<false>, 15ul, 16ul>::reserve<>(unsigned long) (in /data1/lidezhu/tiflash-scripts/integrated/nodes/8284/tiflash1/tiflash/tiflash)
==8154==    by 0xBD4BB66: DB::DataTypeString::deserializeBinaryBulk(DB::IColumn&, DB::ReadBuffer&, unsigned long, double) const (in /data1/lidezhu/tiflash-scripts/integrated/nodes/8284/tiflash1/tiflash/tiflash)
==8154==    by 0x46A0B97: DB::IDataType::deserializeBinaryBulkWithMultipleStreams(DB::IColumn&, std::__1::function<DB::ReadBuffer* (std::__1::vector<DB::IDataType::Substream, std::__1::allocator<DB::IDataType::Substream> > const&)> const&, unsigned long, double, bool, std::__1::vector<DB::IDataType::Substream, std::__1::allocator<DB::IDataType::Substream> >&) const (in /data1/lidezhu/tiflash-scripts/integrated/nodes/8284/tiflash1/tiflash/tiflash)
==8154==    by 0xBD3A4BA: DB::DataTypeNullable::deserializeBinaryBulkWithMultipleStreams(DB::IColumn&, std::__1::function<DB::ReadBuffer* (std::__1::vector<DB::IDataType::Substream, std::__1::allocator<DB::IDataType::Substream> > const&)> const&, unsigned long, double, bool, std::__1::vector<DB::IDataType::Substream, std::__1::allocator<DB::IDataType::Substream> >&) const (in /data1/lidezhu/tiflash-scripts/integrated/nodes/8284/tiflash1/tiflash/tiflash)
==8154==    by 0xBCD1643: DB::IDataType::deserializeBinaryBulkWithMultipleStreams(DB::IColumn&, std::__1::function<DB::ReadBuffer* (std::__1::vector<DB::IDataType::Substream, std::__1::allocator<DB::IDataType::Substream> > const&)> const&, unsigned long, double, bool, std::__1::vector<DB::IDataType::Substream, std::__1::allocator<DB::IDataType::Substream> >&&) const (in /data1/lidezhu/tiflash-scripts/integrated/nodes/8284/tiflash1/tiflash/tiflash)
==8154==    by 0xD708A01: DB::DM::deserializeColumn(DB::IColumn&, std::__1::shared_ptr<DB::IDataType const> const&, DB::ByteBuffer const&, unsigned long) (in /data1/lidezhu/tiflash-scripts/integrated/nodes/8284/tiflash1/tiflash/tiflash)
==8154==    by 0xD6FDC70: DB::DM::ColumnFileTiny::readFromDisk(DB::PageReader const&, std::__1::vector<DB::DM::ColumnDefine, std::__1::allocator<DB::DM::ColumnDefine> > const&, unsigned long, unsigned long) const (in /data1/lidezhu/tiflash-scripts/integrated/nodes/8284/tiflash1/tiflash/tiflash)
==8154==    by 0xD6FDEB8: DB::DM::ColumnFileTiny::fillColumns(DB::PageReader const&, std::__1::vector<DB::DM::ColumnDefine, std::__1::allocator<DB::DM::ColumnDefine> > const&, unsigned long, std::__1::vector<COWPtr<DB::IColumn>::immutable_ptr<DB::IColumn>, std::__1::allocator<COWPtr<DB::IColumn>::immutable_ptr<DB::IColumn> > >&) const (in /data1/lidezhu/tiflash-scripts/integrated/nodes/8284/tiflash1/tiflash/tiflash)
==8154==

Diff with read(flied) between v2 and v3 is that v3 will direct use total entry.size as buffer.

So here is the situation:

In V3 we got an entry.size buffer when we use read(flied) which is shorter than V2(If we don't need to get all of fileds).
Then we call the getFieldData in Page.h. Once index is size() - 1 Then the buffer will longer than filed size.
Then in ColumnFileTiny.cpp, we will use buffer length to deserialize the buffer. and it will cause heap error.

For example:

A filedoffsets [10,20,30,40]
Call read(filed_index) [0,2]
In V3, we got a buffer [10, 30, 60]
In V2, we got a buffer [10, 30]
And when we call `getFieldData` in `Page.h` by index 2
from  buffer [10, 30, 60] got a WriteBuferr [30,60] , length is 90
from  buffer [10, 30] got a WriteBuferr [30] , length is 30
Then in `ColumnFileTiny.cpp`,  we will use buffer length to deserialize the buffer. and it will cause `heap error`.

What is changed and how it works?

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

None

Signed-off-by: jiaqizho <[email protected]>

ti-chi-bot · 2022-04-18T07:14:43Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

JaySon-Huang
lidezhu

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

Signed-off-by: jiaqizho <[email protected]>

dbms/src/Storages/Page/V3/BlobStore.cpp

dbms/src/Storages/Page/V3/PageEntry.h

dbms/src/Storages/Page/V3/tests/gtest_blob_store.cpp

dbms/src/Storages/Page/V3/BlobStore.cpp

JaySon-Huang

LGTM

JaySon-Huang · 2022-04-21T04:58:48Z

Suggest also making enable_ps_v3 default to be true for testing it via our daily/QA test case

tiflash/dbms/src/Server/StorageConfigParser.cpp

Lines 205 to 209 in ebb28dc

    
           // config for experimental feature, may remove later 
        
           if (auto enable_v3 = table->get_qualified_as<Int32>("enable_ps_v3"); enable_v3) 
        
           { 
        
               enable_ps_v3 = (*enable_v3 != 0); 
        
           }

jiaqizho · 2022-04-21T06:18:36Z

Suggest also making enable_ps_v3 default to be true for testing it via our daily/QA test case

tiflash/dbms/src/Server/StorageConfigParser.cpp

Lines 205 to 209 in ebb28dc

// config for experimental feature, may remove later

if (auto enable_v3 = table->get_qualified_as<Int32>("enable_ps_v3"); enable_v3)

{

enable_ps_v3 = (*enable_v3 != 0);

}

@JaySon-Huang

Do we have some macro to detect the daily/QA test ?

If we changed the default enable_ps_v3 to 1, Then the new development Tiflash will use V3. Maybe some of users will be confused that we enable an experimental feature?

Or can we just config the setting in CI?

Signed-off-by: jiaqizho <[email protected]>

jiaqizho · 2022-04-21T09:13:52Z

b70b8c6 PTAL this change which used to open v3 by default.
@JaySon-Huang @lidezhu

JaySon-Huang

LGTM

jiaqizho · 2022-04-21T10:18:38Z

/merge

ti-chi-bot · 2022-04-21T10:18:39Z

@jiaqizho: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2022-04-21T10:18:41Z

This pull request has been accepted and is ready to merge.

Commit hash: 5e4a049

sre-bot · 2022-04-21T10:40:58Z

Coverage for changed files

Filename                                          Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Server/StorageConfigParser.cpp                        407                91    77.64%          22                 1    95.45%         411                26    93.67%         238                87    63.45%
Storages/Page/PageDefines.h                             8                 2    75.00%           8                 2    75.00%          14                 4    71.43%           0                 0         -
Storages/Page/V3/BlobStore.cpp                        480               162    66.25%          49                 5    89.80%         993               268    73.01%         300               124    58.67%
Storages/Page/V3/LogFile/LogWriter.cpp                 50                 1    98.00%           8                 1    87.50%         109                 8    92.66%          32                 1    96.88%
Storages/Page/V3/PageDirectory.cpp                    504                78    84.52%          31                 2    93.55%        1063               187    82.41%         394                93    76.40%
Storages/Page/V3/PageEntriesEdit.h                     54                10    81.48%          39                 8    79.49%         159                44    72.33%          14                 6    57.14%
Storages/Page/V3/PageEntry.h                           16                 2    87.50%           4                 0   100.00%          28                 8    71.43%           8                 2    75.00%
Storages/Page/V3/tests/gtest_blob_store.cpp          4100               710    82.68%          22                 0   100.00%         899                 7    99.22%        1280               604    52.81%
Storages/Page/V3/tests/gtest_page_storage.cpp        2730              1007    63.11%          33                 7    78.79%         751               154    79.49%         868               517    40.44%
Storages/Page/WriteBatch.h                             35                 5    85.71%          21                 2    90.48%         125                22    82.40%          22                 4    81.82%
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                8384              2068    75.33%         237                28    88.19%        4552               728    84.01%        3156              1438    54.44%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
17137      9492             44.61%    193010  96306        50.10%

full coverage report (for internal network access only)

sre-bot · 2022-04-21T10:43:26Z

Coverage for changed files

Filename                                          Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Server/StorageConfigParser.cpp                        407                91    77.64%          22                 1    95.45%         411                26    93.67%         238                87    63.45%
Storages/Page/PageDefines.h                             8                 2    75.00%           8                 2    75.00%          14                 4    71.43%           0                 0         -
Storages/Page/V3/BlobStore.cpp                        480               162    66.25%          49                 5    89.80%         993               268    73.01%         300               124    58.67%
Storages/Page/V3/LogFile/LogWriter.cpp                 50                 1    98.00%           8                 1    87.50%         109                 8    92.66%          32                 1    96.88%
Storages/Page/V3/PageDirectory.cpp                    504                78    84.52%          31                 2    93.55%        1063               187    82.41%         394                94    76.14%
Storages/Page/V3/PageEntriesEdit.h                     54                10    81.48%          39                 8    79.49%         159                44    72.33%          14                 6    57.14%
Storages/Page/V3/PageEntry.h                           16                 2    87.50%           4                 0   100.00%          28                 8    71.43%           8                 2    75.00%
Storages/Page/V3/tests/gtest_blob_store.cpp          4100               710    82.68%          22                 0   100.00%         899                 7    99.22%        1280               604    52.81%
Storages/Page/V3/tests/gtest_page_storage.cpp        2730              1007    63.11%          33                 7    78.79%         751               154    79.49%         868               517    40.44%
Storages/Page/WriteBatch.h                             35                 5    85.71%          21                 2    90.48%         125                22    82.40%          22                 4    81.82%
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                8384              2068    75.33%         237                28    88.19%        4552               728    84.01%        3156              1439    54.40%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
17147      9494             44.63%    193112  96304        50.13%

full coverage report (for internal network access only)

sre-bot · 2022-05-24T10:16:26Z

Coverage for changed files

Filename                                          Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Server/StorageConfigParser.cpp                        407                91    77.64%          22                 1    95.45%         411                26    93.67%         238                87    63.45%
Storages/Page/PageDefines.h                             8                 2    75.00%           8                 2    75.00%          14                 4    71.43%           0                 0         -
Storages/Page/V3/BlobStore.cpp                        480               162    66.25%          49                 5    89.80%         993               268    73.01%         300               124    58.67%
Storages/Page/V3/LogFile/LogWriter.cpp                 50                 1    98.00%           8                 1    87.50%         109                 8    92.66%          32                 1    96.88%
Storages/Page/V3/PageDirectory.cpp                    504                78    84.52%          31                 2    93.55%        1063               187    82.41%         394                94    76.14%
Storages/Page/V3/PageEntriesEdit.h                     54                10    81.48%          39                 8    79.49%         159                44    72.33%          14                 6    57.14%
Storages/Page/V3/PageEntry.h                           16                 2    87.50%           4                 0   100.00%          28                 8    71.43%           8                 2    75.00%
Storages/Page/V3/tests/gtest_blob_store.cpp          4100               710    82.68%          22                 0   100.00%         899                 7    99.22%        1280               604    52.81%
Storages/Page/V3/tests/gtest_page_storage.cpp        2730              1007    63.11%          33                 7    78.79%         751               154    79.49%         868               517    40.44%
Storages/Page/WriteBatch.h                             35                 5    85.71%          21                 2    90.48%         125                22    82.40%          22                 4    81.82%
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                8384              2068    75.33%         237                28    88.19%        4552               728    84.01%        3156              1439    54.40%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
17147      9494             44.63%    193112  96304        50.13%

full coverage report (for internal network access only)

add tests and op

6c0f2ff

Signed-off-by: jiaqizho <[email protected]>

ti-chi-bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note-none Denotes a PR that doesn't merit a release note. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Apr 18, 2022

jiaqizho added 2 commits April 19, 2022 17:31

fi

3e4c557

Merge branch 'master' into fix-ddl-error

e292d65

ti-chi-bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Apr 19, 2022

jiaqizho added 3 commits April 19, 2022 18:38

update

c20d621

Signed-off-by: jiaqizho <[email protected]>

change filed read

feac128

update

d2df38d

jiaqizho changed the title ~~[WIP]Try to fix pagestorage v3 ddl problem.~~ Fix pagestorage v3 ddl problem. Apr 20, 2022

ti-chi-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 20, 2022

jiaqizho requested review from JaySon-Huang, lidezhu and flowbehappy April 20, 2022 01:51

Merge branch 'master' into fix-ddl-error

0af197d

JaySon-Huang reviewed Apr 20, 2022

View reviewed changes

dbms/src/Storages/Page/V3/BlobStore.cpp Show resolved Hide resolved

dbms/src/Storages/Page/V3/BlobStore.cpp Show resolved Hide resolved

dbms/src/Storages/Page/V3/PageEntry.h Outdated Show resolved Hide resolved

update

2bda7ce

JaySon-Huang reviewed Apr 20, 2022

View reviewed changes

update

1b7895e

JaySon-Huang approved these changes Apr 21, 2022

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Apr 21, 2022

lidezhu approved these changes Apr 21, 2022

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Apr 21, 2022

open v3 by default

b70b8c6

Signed-off-by: jiaqizho <[email protected]>

JaySon-Huang approved these changes Apr 21, 2022

View reviewed changes

1 to true

5e4a049

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Apr 21, 2022

Merge branch 'master' into fix-ddl-error

91ad8ef

ti-chi-bot merged commit d1e2584 into pingcap:master Apr 21, 2022

lidezhu mentioned this pull request May 24, 2022

TiFlash crash when running schrodinger/ddl #4712

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix pagestorage v3 ddl problem. #4691

Fix pagestorage v3 ddl problem. #4691

jiaqizho commented Apr 18, 2022 •

edited by lidezhu

Loading

ti-chi-bot commented Apr 18, 2022 •

edited

Loading

JaySon-Huang left a comment

JaySon-Huang commented Apr 21, 2022 •

edited

Loading

jiaqizho commented Apr 21, 2022 •

edited

Loading

jiaqizho commented Apr 21, 2022

JaySon-Huang left a comment

jiaqizho commented Apr 21, 2022

ti-chi-bot commented Apr 21, 2022

ti-chi-bot commented Apr 21, 2022

sre-bot commented Apr 21, 2022

sre-bot commented Apr 21, 2022

sre-bot commented May 24, 2022

Fix pagestorage v3 ddl problem. #4691

Fix pagestorage v3 ddl problem. #4691

Conversation

jiaqizho commented Apr 18, 2022 • edited by lidezhu Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Apr 18, 2022 • edited Loading

JaySon-Huang left a comment

Choose a reason for hiding this comment

JaySon-Huang commented Apr 21, 2022 • edited Loading

jiaqizho commented Apr 21, 2022 • edited Loading

jiaqizho commented Apr 21, 2022

JaySon-Huang left a comment

Choose a reason for hiding this comment

jiaqizho commented Apr 21, 2022

ti-chi-bot commented Apr 21, 2022

ti-chi-bot commented Apr 21, 2022

sre-bot commented Apr 21, 2022

Coverage for changed files

Coverage summary

sre-bot commented Apr 21, 2022

Coverage for changed files

Coverage summary

sre-bot commented May 24, 2022

Coverage for changed files

Coverage summary

jiaqizho commented Apr 18, 2022 •

edited by lidezhu

Loading

ti-chi-bot commented Apr 18, 2022 •

edited

Loading

JaySon-Huang commented Apr 21, 2022 •

edited

Loading

jiaqizho commented Apr 21, 2022 •

edited

Loading