The block decoded from SSTFile is not sorted by primary key and version

[lilinghai]

Maybe the mini reproduce:

1. prepare TPCC 10k warehouses and set tiflash replica 2
2. after the replication is down, set raft.snapshot.method: file1 and stop the tiflash nodes
3. execute the dml sqls

insert into stock select s_i_id,10003,s_quantity,s_dist_01,s_dist_02,s_dist_03,s_dist_04,s_dist_05,s_dist_06,s_dist_07,s_dist_08,s_dist_09,s_dist_10,s_ytd,s_order_cnt,s_remote_cnt,s_data from stock where s_w_id=1;
delete from stock where s_w_id>=200 and s_w_id<=220;
update stock set s_order_cnt=s_order_cnt+1 where s_w_id>=1 and s_w_id<=100;

4. start tiflash nodes
   The tiflash node can't start successfully and after 20 mins the tiflash nodes start. The error log is

2021.05.06 16:08:55.621690 [ 10 ] <Error> DB::RawCppPtr DB::PreHandleSnapshot(DB::EngineStoreServerWrap*, DB::BaseBuffView, uint64_t, DB::SSTViewVec, uint64_t, uint64_t): Code: 0, e.displayText() = DB::Exception: The block decoded from SSTFile is not sorted by primary key and version [region=[region 127392, applied: term 7 index 1242]], e.what() = DB::Exception, Stack trace:

0. bin/tiflash/tiflash(StackTrace::StackTrace()+0x15) [0x366b045]
1. bin/tiflash/tiflash(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)+0x25) [0x3661bd5]
2. bin/tiflash/tiflash(DB::DM::SSTFilesToDTFilesOutputStream::write()+0x490) [0x77db1c0]
3. bin/tiflash/tiflash(DB::KVStore::preHandleSSTsToDTFiles(std::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, DB::DM::FileConvertJobType, DB::TMTContext&)+0x36e) [0x769210e]
4. bin/tiflash/tiflash(DB::KVStore::preHandleSnapshotToFiles(std::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, DB::TMTContext&)+0x45) [0x7692865]
5. bin/tiflash/tiflash(DB::PreHandleSnapshot(DB::EngineStoreServerWrap*, DB::BaseBuffView, unsigned long, DB::SSTViewVec, unsigned long, unsigned long)+0xf3) [0x72db683]
6. 2021.05.06 16:08:55.621690 [ 10 ] <Error> DB::RawCppPtr DB::PreHandleSnapshot(DB::EngineStoreServerWrap*, DB::BaseBuffView, uint64_t, DB::SSTViewVec, uint64_t, uint64_t): Code: 0, e.displayText() = DB::Exception: The block decoded from SSTFile is not sorted by primary key and version [region=[region 127392, applied: term 7 index 1242]], e.what() = DB::Exception, Stack trace:

0. bin/tiflash/tiflash(StackTrace::StackTrace()+0x15) [0x366b045]
1. bin/tiflash/tiflash(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)+0x25) [0x3661bd5]
2. bin/tiflash/tiflash(DB::DM::SSTFilesToDTFilesOutputStream::write()+0x490) [0x77db1c0]
3. bin/tiflash/tiflash(DB::KVStore::preHandleSSTsToDTFiles(std::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, DB::DM::FileConvertJobType, DB::TMTContext&)+0x36e) [0x769210e]
4. bin/tiflash/tiflash(DB::KVStore::preHandleSnapshotToFiles(std::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, DB::TMTContext&)+0x45) [0x7692865]
5. bin/tiflash/tiflash(DB::PreHandleSnapshot(DB::EngineStoreServerWrap*, DB::BaseBuffView, unsigned long, DB::SSTViewVec, unsigned long, unsigned long)+0xf3) [0x72db683]

[JaySon-Huang]

The root cause is located.

The rows (primary key, timestamp) are sorted by primary key asc, timestamp desc in SSTFiles. But we mistook they are sorted by primary key asc, timestamp asc.
While the rows are sorted by primary key asc, timestamp asc in TiFlash's DTFile. We need to take care of it on the TiFlash side.

I will file a PR to fix it.

Reference:

• MVCC in TiKV
• The encoded key is sorted by pk asc, timestamp desc in TiKV: types.rs