Recommend/convert to use case-sensitive collation for MySQL/MSSQL (#2…

…8662)

Mainly for MySQL/MSSQL.

It is important for Gitea to use case-sensitive database charset
collation. If the database is using a case-insensitive collation, Gitea
will show startup error/warning messages, and show the errors/warnings
on the admin panel's Self-Check page.

Make `gitea doctor convert` work for MySQL to convert the collations of
database & tables & columns.

* Fix #28131

## ⚠️ BREAKING ⚠️

It is not quite breaking, but it's highly recommended to convert the
database&table&column to a consistent and case-sensitive collation.

.gitignore

@@ -11,6 +11,7 @@ _test
 .idea
 # Goland's output filename can not be set manually
 /go_build_*
+/gitea_*
 
 # MS VSCode
 .vscode

cmd/doctor_convert.go

@@ -37,8 +37,8 @@ func runDoctorConvert(ctx *cli.Context) error {
 
 	switch {
 	case setting.Database.Type.IsMySQL():
-		if err := db.ConvertUtf8ToUtf8mb4(); err != nil {
-			log.Fatal("Failed to convert database from utf8 to utf8mb4: %v", err)
+		if err := db.ConvertDatabaseTable(); err != nil {
+			log.Fatal("Failed to convert database & table: %v", err)
 			return err
 		}
 		fmt.Println("Converted successfully, please confirm your database's character set is now utf8mb4")

custom/conf/app.example.ini

@@ -351,6 +351,7 @@ NAME = gitea
 USER = root
 ;PASSWD = ;Use PASSWD = `your password` for quoting if you use special characters in the password.
 ;SSL_MODE = false ; either "false" (default), "true", or "skip-verify"
+;CHARSET_COLLATION = ; Empty as default, Gitea will try to find a case-sensitive collation. Don't change it unless you clearly know what you need.
 ;;
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 ;;
@@ -382,6 +383,7 @@ USER = root
 ;NAME = gitea
 ;USER = SA
 ;PASSWD = MwantsaSecurePassword1
+;CHARSET_COLLATION = ; Empty as default, Gitea will try to find a case-sensitive collation. Don't change it unless you clearly know what you need.
 ;;
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 ;;

docs/content/administration/config-cheat-sheet.en-us.md

@@ -430,6 +430,7 @@ The following configuration set `Content-Type: application/vnd.android.package-a
 - `NAME`: **gitea**: Database name.
 - `USER`: **root**: Database username.
 - `PASSWD`: **_empty_**: Database user password. Use \`your password\` or """your password""" for quoting if you use special characters in the password.
+- `CHARSET_COLLATION`: **_empty_**: (MySQL/MSSQL only) Gitea expects to use a case-sensitive collation for database. Leave it empty to use the default collation decided by the Gitea. Don't change it unless you clearly know what you need.
 - `SCHEMA`: **_empty_**: For PostgreSQL only, schema to use if different from "public". The schema must exist beforehand,
   the user must have creation privileges on it, and the user search path must be set to the look into the schema first
   (e.g. `ALTER USER user SET SEARCH_PATH = schema_name,"$user",public;`).

docs/content/help/faq.en-us.md

@@ -371,24 +371,6 @@ If you are receiving an error line containing `Error 1071: Specified key was too
 then you are attempting to run Gitea on tables which use the ISAM engine. While this may have worked by chance in previous versions of Gitea, it has never been officially supported and
 you must use InnoDB. You should run `ALTER TABLE table_name ENGINE=InnoDB;` for each table in the database.
 
-If you are using MySQL 5, another possible fix is
-
-```mysql
-SET GLOBAL innodb_file_format=Barracuda;
-SET GLOBAL innodb_file_per_table=1;
-SET GLOBAL innodb_large_prefix=1;
-```
-
-## Why Are Emoji Broken On MySQL
-
-Unfortunately MySQL's `utf8` charset does not completely allow all possible UTF-8 characters, in particular Emoji.
-They created a new charset and collation called `utf8mb4` that allows for emoji to be stored but tables which use
-the `utf8` charset, and connections which use the `utf8` charset will not use this.
-
-Please run `gitea doctor convert`, or run `ALTER DATABASE database_name CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;`
-for the database_name and run `ALTER TABLE table_name CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;`
-for each table in the database.
-
 ## Why are Emoji displaying only as placeholders or in monochrome
 
 Gitea requires the system or browser to have one of the supported Emoji fonts installed, which are Apple Color Emoji, Segoe UI Emoji, Segoe UI Symbol, Noto Color Emoji and Twemoji Mozilla. Generally, the operating system should already provide one of these fonts, but especially on Linux, it may be necessary to install them manually.

docs/content/help/faq.zh-cn.md

@@ -375,25 +375,6 @@ Gitea 提供了一个子命令`gitea migrate`来初始化数据库，然后您
 的错误行，则表示您正在尝试在使用 ISAM 引擎的表上运行 Gitea。尽管在先前版本的 Gitea 中可能是凑巧能够工作的，但它从未得到官方支持，
 您必须使用 InnoDB。您应该对数据库中的每个表运行`ALTER TABLE table_name ENGINE=InnoDB;`。
 
-如果您使用的是 MySQL 5，另一个可能的修复方法是：
-
-```mysql
-SET GLOBAL innodb_file_format=Barracuda;
-SET GLOBAL innodb_file_per_table=1;
-SET GLOBAL innodb_large_prefix=1;
-```
-
-## 为什么 MySQL 上的 Emoji 显示错误
-
-不幸的是，MySQL 的`utf8`字符集不完全允许所有可能的 UTF-8 字符，特别是 Emoji。
-他们创建了一个名为 `utf8mb4`的字符集和校对规则，允许存储 Emoji，但使用
-utf8 字符集的表和连接将不会使用它。
-
-请运行 `gitea doctor convert` 或对数据库运行 `ALTER DATABASE database_name CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;`
-并对每个表运行 `ALTER TABLE table_name CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;`。
-
-您还需要将`app.ini`文件中的数据库字符集设置为`CHARSET=utf8mb4`。
-
 ## 为什么 Emoji 只显示占位符或单色图像
 
 Gitea 需要系统或浏览器安装其中一个受支持的 Emoji 字体，例如 Apple Color Emoji、Segoe UI Emoji、Segoe UI Symbol、Noto Color Emoji 和 Twemoji Mozilla。通常，操作系统应该已经提供了其中一个字体，但特别是在 Linux 上，可能需要手动安装它们。

docs/content/installation/database-preparation.en-us.md

@@ -61,10 +61,14 @@ Note: All steps below requires that the database engine of your choice is instal
 
     Replace username and password above as appropriate.
 
-4. Create database with UTF-8 charset and collation. Make sure to use `utf8mb4` charset instead of `utf8` as the former supports all Unicode characters (including emojis) beyond _Basic Multilingual Plane_. Also, collation chosen depending on your expected content. When in doubt, use either `unicode_ci` or `general_ci`.
+4. Create database with UTF-8 charset and case-sensitive collation.
+
+    `utf8mb4_bin` is a common collation for both MySQL/MariaDB.
+    When Gitea starts, it will try to find a better collation (`utf8mb4_0900_as_cs` or `uca1400_as_cs`) and alter the database if it is possible.
+    If you would like to use other collation, you can set `[database].CHARSET_COLLATION` in the `app.ini` file.
 
     ```sql
-    CREATE DATABASE giteadb CHARACTER SET 'utf8mb4' COLLATE 'utf8mb4_unicode_ci';
+    CREATE DATABASE giteadb CHARACTER SET 'utf8mb4' COLLATE 'utf8mb4_bin';
     ```
 
     Replace database name as appropriate.

docs/content/installation/database-preparation.zh-cn.md

@@ -59,10 +59,12 @@ menu:
 
     根据需要替换上述用户名和密码。
 
-4. 使用 UTF-8 字符集和排序规则创建数据库。确保使用 `**utf8mb4**` 字符集，而不是 `utf8`，因为前者支持 _Basic Multilingual Plane_ 之外的所有 Unicode 字符（包括表情符号）。排序规则根据您预期的内容选择。如果不确定，可以使用 `unicode_ci` 或 `general_ci`。
+4. 使用 UTF-8 字符集和大小写敏感的排序规则创建数据库。
+
+    Gitea 启动后会尝试把数据库修改为更合适的字符集，如果你想指定自己的字符集规则，可以在 app.ini 中设置 `[database].CHARSET_COLLATION`。
 
     ```sql
-    CREATE DATABASE giteadb CHARACTER SET 'utf8mb4' COLLATE 'utf8mb4_unicode_ci';
+    CREATE DATABASE giteadb CHARACTER SET 'utf8mb4' COLLATE 'utf8mb4_bin';
     ```
 
     根据需要替换数据库名称。

go.mod

@@ -121,7 +121,7 @@ require (
 	mvdan.cc/xurls/v2 v2.5.0
 	strk.kbt.io/projects/go/libravatar v0.0.0-20191008002943-06d1c002b251
 	xorm.io/builder v0.3.13
-	xorm.io/xorm v1.3.6
+	xorm.io/xorm v1.3.7-0.20240101024435-4992cba040fe
 )
 
 require (