Using certain unicode characters in a test case causes a "500 Service Error"

[KiraResari]

Description of problem

Using certain unicode characters in a test case causes a "500 Service Error"

Component (web, API, etc)

Web Portal

Version or commit hash (if applicable)

4.1.3

How often reproducible

100%

Steps to Reproduce

1. Create a new test case
2. Enter the unicode character "🔍" in any field (tested for "Summary", "Notes", "Setup", "Breakdown", "Actions" and "Expected Results".
3. Click the [Save]-button

Actual results

A "500 Service Error" page is displayed

Expected results

The test case should be created as normal

Additional info

[atodorov]

Thanks for the bug report.

The screenshot is useless since it doesn't give me any information as to where the error is.

Try to reproduce this on the live demo site, it will automatically report the error to us with all the details that we need.

Also give the unicode character code b/c I only see the emoji icon and I'm not going to spend time to search for this in the entire unicode table.

[KiraResari]

Left-Pointing Magnifying Glass U+1F50D
Took me 35 seconds to find it using Bing (searched for "🔍 unicode")
Also see https://unicode-table.com/de/1F50D/

Anyway, I could successfully reproduce this on the life demo site as well on first try.
Have fun =^,^=

[sarzamas]

@atodorov
I have the same problem with HTTP 500 on my local kiwi tcms dev server (docker install with MariaDB engine)
Kiwi TCMS ver 8.1.99 (04 Mar 2020)

• When I use Russian chars in create user/test/bug forms on my server - after pushing save button I see HTTP 500
• When I do the same on public demo server - it works fine - no error

What was done to investigate:

If I set DEBUG = true
I receive following trace buffer in attachment
HTTP500 for russian charset.txt

Exception Type: OperationalError at /cases/new/
Exception Value: (1366, "Incorrect string value: '\\xD0\\xBA\\xD0\\xBA\\xD0\\xB5' for column `kiwi`.`testcases_testcase`.`summary` at row 1")

Execute troubleshooting according to this help:
https://kiwitcms.readthedocs.io/en/latest/changelog.html#oct-09-2017-released-on-mrsenko-com

With some addition from this help: uft8 replaced with utf8mb4
https://stackoverflow.com/questions/202205/how-to-make-mysql-handle-utf-8-properly

# docker exec --privileged -it kiwi_db bash

bash-4.2$ mysql -D kiwi -u root --password= -B -N -e "SHOW TABLE STATUS"
bash-4.2$ mysql -D kiwi -u root --password= -B -N -e "ALTER DATABASE kiwi CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci"
bash-4.2$ mysql -D kiwi -u root --password= -B -N -e "SHOW TABLES" | awk '{print "ALTER TABLE", $1, "CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;"}' > /tmp/alter_charset.txt
bash-4.2$ cat /tmp/alter_charset.txt | mysql -D kiwi -u root --password=
bash-4.2$ mysql -D kiwi -u root --password= -B -N -e "SHOW TABLE STATUS"

and finally, update common.py according to this Django ticket:
https://code.djangoproject.com/ticket/18392#comment:11

edit file
$ nano ~/Kiwi/tcms/settings/common.py
in section:

handle MariaDB only options

if DATABASES['default']['ENGINE'].find('mysql') > -1:
    DATABASES['default']['OPTIONS'].update({  # pylint: disable=objects-update-used
        'init_command': "SET sql_mode='STRICT_TRANS_TABLES'",
        'charset': 'utf8mb4',
        })

and after that make new docker image with MariaDB customization

up containers and all works fine now with any unicode characters like 🔍

[sarzamas]

@atodorov
ticket may be closed

as it is configuration issue

maybe need to update documentation... cause I spent a day to find out a reason of HTTP500 using emoji in forms

[atodorov]

@sarzamas - please send a pull request for documentation updates where you think is unclear instructions.

Also what language is that, which requires the utf8mb4 charset ? We should probably document this as well.

[sarzamas]

@atodorov

for language charset support in forms - utf8 is sufficient
for emoji support in forms - utf8mb4 is necessary

Regarding pull request... there is nothing on this theme in docs yet except this ticket
my steps in history for a solution:

1. while installation KIWI local server from dockerhub image :latest on initial step create new user I immediately faced HTTP500
   TIP: using english (ASCII chars) in forms it worked, but not my native language chars!
2. Then I set debug flag to check the backtrace
3. reading log I realized the problem with MariaDB not parcing localized chars
4. Searching KIWI site help gave no result on this matter (using keyword localization)
5. Then I found this particular issue and checked public KIWI - bug is not reproducible there!!!
6. Next steps to solve I described in my previous post

I believe there should be new wiki page describing a configuration of localization/emodji support in mariaDB instances (ust using my steps above) - and link to this page in INSTALLATION /SETUP page https://kiwitcms.readthedocs.io/en/latest/installing_docker.html#initial-configuration-of-running-container

[atodorov]

@sarzamas - public.tenant is hosted with Postgres (utf8 charset) and emojis seem to be supported:
https://public.tenant.kiwitcms.org/case/8579/

I have changed the default charset & collation for MariaDB to utf8mb4 and it doesn't break any existing tests so I think it is safe to leave it as default.

Also added a short section in the configuration page to point out this may be something an admin would like to look at. It really depends on what you language is and how your DB server is configured (also what languages does it support) so it's hard to give any more specific information.

[deltoro05]

Left-Pointing Magnifying Glass U+1F50D Took me 35 seconds to find it using Bing (searched for "🔍 unicode") Also see https://unicode-table.com/de/1F50D/

Anyway, I could successfully reproduce this on the life demo site as well on first try. Have fun =^,^=

This URL has changed to https://symbl.cc/de/1F50D/, it will be cool if you fix it )

[deltoro05]

Left-Pointing Magnifying Glass U+1F50D Took me 35 seconds to find it using Bing (searched for "🔍 unicode") Also see https://unicode-table.com/de/1F50D/

Anyway, I could successfully reproduce this on the life demo site as well on first try. Have fun =^,^=

This URL https://unicode-table.com/ has changed to https://symbl.cc/