Support GBK

[zimulala]

Description

Currently, TiDB only supports ASCII, binary, latin1, UTF8, and UTF8MB4 character sets. This proposal aims to add a framework for character sets to TiDB to facilitate the support of other characters. This issue is used to support the GBK character set.
This issue tracks our progress in supporting GBK.

• Weekly Report
• Proposal: Support charsets in TiDB

Sub Tasks

Stage 1

Encoder/Decoder

• Implement GBK encode/decode util functions #26813

Parser

• Provide a parse interface to accept charset_client config #26814
• Set character_set_client for parser properly #26815

Runtime

• Support GBK for string-related built-in functions #26816
• On the Coprocessor side, most string-related builtin functions need special processing after conversion from utf8 #26817

Optimizer @xiongjiwei

• Special processing of GBK character set during ranger processing #26820
• Consider GBK character set when dealing with statistics information related issues #26819

Collation @xiongjiwei

• make gbk_bin get supported in infer collation #27331
• Support gbk_bin
  • Support gbk_bin on TiDB #26821
  • Support gbk_bin on Coprocessor #26822

DDL @xiongjiwei

• Support for database, table, column character set when building database, table or adding column #26823

Others

• Adding an implicit switch to facilitate functionality before GA  #26834
  • @xiongjiwei
• Support setting character set operations, such as set character set gbk, set names gbk, set character_set_client = gbk, etc. #26824
  • @xiongjiwei
• Support correct display character set operations, such as show charset/collation #26825
• support translate query results to character set result before shipping them back to the client #27720
• Check for invalid GBK characters before insertion #28778
• return error if SQL contatins invalid character under strict mode #29382

Stage 2

Collation @xiongjiwei

• gbk_chinese_ci collation support #28248

Compatibility with other components

• TiFlash related: Forbid set TiFlashReplica for a table with GBK charset #30095
• Lightning: parsing the SQL into KV and using functions like CastValue internally requires some processing
• DM: handling of GBK character set when connecting to TiDB and when parsing DDL https://github.com/pingcap/dm/issues/2226
• TiDB-binlog: Handlers need to be converted to GBK character set encoding when data is exported downstream.
• Removing this function‘s switch #31686

Test

• Porting GBK character set and related collation tests from mysql tests. #26831
• Correctness and system test
• Compatibility test with other components
• Performance test

Stage 3

• TiCDC: When outputting downstream, it needs to be converted to GBK character set encoding.
• TiFlash supports it.
  • TiFlash: most string-related builtin functions need special processing after conversion from utf8
  • TiFlash: support gbk_bin
  • TiFlash: support gbk_chinese_ci
• Basically all string related functions are supported
• GBK: Support the use of ALTER statements, support to change the charset of columns. #28785
• Make a case for supporting other characters, such as the GB 18030 character set #31688

[tisonkun]

@zimulala I can see that several task items marked as done but the issue is still open, why?

[zimulala]

@tisonkun
Jiwei's PR dealt with multiple issues before, and he may not have noticed that the corresponding issues were not closed. It's done now.