Kiwi 8.4 does not support Cyrillic

[DMyachin]

1. Create new test case with Title тест
2. Save

Expected: test case created
Actual: error 500. See log below:

Environment:

Request Method: POST
Request URL: https://kiwi.tacita.net/cases/new/

Django Version: 3.0.7
Python Version: 3.6.8
Installed Applications:
['django.contrib.sites',
'attachments',
'django_comments',
'modernrpc',
'simple_history',
'tcms.core.contrib.linkreference',
'tcms.management',
'tcms.testcases.apps.AppConfig',
'tcms.testplans.apps.AppConfig',
'tcms.testruns.apps.AppConfig',
'tcms.bugs.apps.AppConfig',
'grappelli',
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.messages',
'django.contrib.sessions',
'django.contrib.staticfiles',
'colorfield',
'vinaigrette',
'tcms.core',
'tcms.kiwi_auth',
'tcms.telemetry',
'tcms.rpc']
Installed Middleware:
['django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.locale.LocaleMiddleware',
'django.middleware.common.CommonMiddleware',
'tcms.core.middleware.CsrfDisableMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'global_login_required.GlobalLoginRequiredMiddleware',
'simple_history.middleware.HistoryRequestMiddleware',
'tcms.core.middleware.CheckSettingsMiddleware',
'vinaigrette.middleware.VinaigretteAdminLanguageMiddleware']

Traceback (most recent call last):
File "/venv/lib64/python3.6/site-packages/django/db/backends/utils.py", line 86, in _execute
return self.cursor.execute(sql, params)
File "/venv/lib64/python3.6/site-packages/django/db/backends/mysql/base.py", line 74, in execute
return self.cursor.execute(query, args)
File "/venv/lib64/python3.6/site-packages/MySQLdb/cursors.py", line 209, in execute
res = self._query(query)
File "/venv/lib64/python3.6/site-packages/MySQLdb/cursors.py", line 315, in _query
db.query(q)
File "/venv/lib64/python3.6/site-packages/MySQLdb/connections.py", line 239, in query
_mysql.connection.query(self, query)

The above exception ((1366, "Incorrect string value: '\xD1\x82\xD0\xB5\xD1\x81...' for column kiwi.testcases_testcase.summary at row 1")) was the direct cause of the following exception:
File "/venv/lib64/python3.6/site-packages/django/core/handlers/exception.py", line 34, in inner
response = get_response(request)
File "/venv/lib64/python3.6/site-packages/django/core/handlers/base.py", line 115, in _get_response
response = self.process_exception_by_middleware(e, request)
File "/venv/lib64/python3.6/site-packages/django/core/handlers/base.py", line 113, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/venv/lib64/python3.6/site-packages/django/views/generic/base.py", line 71, in view
return self.dispatch(request, *args, **kwargs)
File "/venv/lib64/python3.6/site-packages/django/utils/decorators.py", line 43, in _wrapper
return bound_method(*args, **kwargs)
File "/venv/lib64/python3.6/site-packages/django/contrib/auth/decorators.py", line 21, in _wrapped_view
return view_func(request, *args, **kwargs)
File "/venv/lib64/python3.6/site-packages/django/views/generic/base.py", line 97, in dispatch
return handler(request, *args, **kwargs)
File "/venv/lib64/python3.6/site-packages/django/views/generic/edit.py", line 172, in post
return super().post(request, *args, **kwargs)
File "/venv/lib64/python3.6/site-packages/django/views/generic/edit.py", line 142, in post
return self.form_valid(form)
File "/venv/lib64/python3.6/site-packages/tcms/testcases/views.py", line 88, in form_valid
test_case = form.save()
File "/venv/lib64/python3.6/site-packages/django/forms/models.py", line 459, in save
self.instance.save()
File "/venv/lib64/python3.6/site-packages/django/db/models/base.py", line 746, in save
force_update=force_update, update_fields=update_fields)
File "/venv/lib64/python3.6/site-packages/django/db/models/base.py", line 784, in save_base
force_update, using, update_fields,
File "/venv/lib64/python3.6/site-packages/django/db/models/base.py", line 887, in _save_table
results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
File "/venv/lib64/python3.6/site-packages/django/db/models/base.py", line 926, in _do_insert
using=using, raw=raw,
File "/venv/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/venv/lib64/python3.6/site-packages/django/db/models/query.py", line 1204, in _insert
return query.get_compiler(using=using).execute_sql(returning_fields)
File "/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1392, in execute_sql
cursor.execute(sql, params)
File "/venv/lib64/python3.6/site-packages/django/db/backends/utils.py", line 100, in execute
return super().execute(sql, params)
File "/venv/lib64/python3.6/site-packages/django/db/backends/utils.py", line 68, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/venv/lib64/python3.6/site-packages/django/db/backends/utils.py", line 77, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/venv/lib64/python3.6/site-packages/django/db/backends/utils.py", line 86, in _execute
return self.cursor.execute(sql, params)
File "/venv/lib64/python3.6/site-packages/django/db/utils.py", line 90, in exit
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/venv/lib64/python3.6/site-packages/django/db/backends/utils.py", line 86, in _execute
return self.cursor.execute(sql, params)
File "/venv/lib64/python3.6/site-packages/django/db/backends/mysql/base.py", line 74, in execute
return self.cursor.execute(query, args)
File "/venv/lib64/python3.6/site-packages/MySQLdb/cursors.py", line 209, in execute
res = self._query(query)
File "/venv/lib64/python3.6/site-packages/MySQLdb/cursors.py", line 315, in _query
db.query(q)
File "/venv/lib64/python3.6/site-packages/MySQLdb/connections.py", line 239, in query
_mysql.connection.query(self, query)

Exception Type: OperationalError at /cases/new/
Exception Value: (1366, "Incorrect string value: '\xD1\x82\xD0\xB5\xD1\x81...' for column kiwi.testcases_testcase.summary at row 1")

[DMyachin]

Python 3.8.3 (default, May 17 2020, 18:15:42) 
[GCC 10.1.0] on linux
Type "help", "copyright", "credits" or "license()" for more information.
>>> 'тест'.encode()
b'\xd1\x82\xd0\xb5\xd1\x81\xd1\x82'
>>> 

[atodorov]

Your database is not configured properly for UTF8 or utf8mb4. MySQL/MariaDB uses latin1 by default which doesn't support Cyrillic while other DB backends have different defaults. The provided docker-compose.yml is configured for utf8mb4 which supports both Cyrillic and even emojis.

You can see that on https://public.tenant.kiwitcms.org Cyrillic is working fine.

For further support questions open a question on StackOverflow as instructed when opening new issues in GitHub.

[DMyachin]

But my docker-compose.yml:

version: '2'
# Myachin: see https://kiwitcms.readthedocs.io/en/latest/installing_docker.html

services:
    db:
        container_name: kiwi_db_messenger
        image: centos/mariadb-103-centos7
        volumes:
            - db_data:/var/lib/mysql/data
            # workaround for missing charset & collation support
            # https://github.com/sclorg/mariadb-container/pull/125
            - ./99-charset.sh:/usr/share/container-scripts/mysql/init/99-charset.sh:Z
            - ./99-charset.cnf:/usr/share/container-scripts/mysql/cnf/99-charset.cnf:Z
        restart: always
        environment:
            MYSQL_ROOT_PASSWORD: kiwi-1s-aw3s0m3
            MYSQL_DATABASE: kiwi
            MYSQL_USER: kiwi
            MYSQL_PASSWORD: kiwi
            MYSQL_CHARSET: utf8mb4
            MYSQL_COLLATION: utf8mb4_unicode_ci
    web:
        container_name: kiwi_web_messenger
        depends_on:
            - db
        restart: always
        image: kiwitcms/kiwi:latest
        ports:
            - 8443:8443
        volumes:
            - uploads:/Kiwi/uploads:Z
            - ./local_settings.py:/venv/lib/python3.6/site-packages/tcms/settings/local_settings.py
        environment:
            KIWI_DB_HOST: db
            KIWI_DB_PORT: 3306
            KIWI_DB_NAME: kiwi
            KIWI_DB_USER: kiwi
            KIWI_DB_PASSWORD: kiwi
            KIWI_DONT_ENFORCE_HTTPS: "true"
volumes:
    db_data:
    uploads:

It's https://raw.githubusercontent.com/kiwitcms/Kiwi/master/docker-compose.yml

And those new Kiwi installation, not upgrade.