Strings Encoding Issues in Python 3 due to latest Cython updates

[madhawav]

The latest release version of Cython (0.27.1) is causing pyjnius issues related to string encoding. I have experienced this issue in Python 3.5 for Ubuntu 16.04 (fresh installation).

The issue is as follows.

A strings passed from Python to Java as parameter of a method gets encoded incorrectly. For example, when I pass the string "query" as a method parameter and system.out.println from Java, it gets printed as "å".

When I install pyjnius using Cython 0.25.2, this issue is resolved.
Also, this issue does not occur with Python 2.7.

---

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[iMcG33k]

The same issue on Cython (0.27.1) Python 3.6 macos 10.13 JDK 1.8.
But java String Works well

Stack = autoclass('java.util.Stack')
stack = Stack()
JString = autoclass('java.lang.String')
stack.push(JString("hello"))
stack.push(JString("world"))
print(stack.pop())

output:
Hello world

world
hello

[psader]

The issue is resolved by modifying populate_args in jnius_conversion.pxi, such that the input py_str is always converted with .encode('utf-8') before calling NewStringUTF. This means only running the code under except (UnicodeEncodeError, TypeError): so that there's only one possible call to NewStringUTF.
The test drivers pass with this change with python 2 and 3. There may be some case not covered by them, but calling a string conversion routine twice to see which one works is a bad code smell, because otherwise you end up with the "Bush hid the facts" bug (https://en.wikipedia.org/wiki/Bush_hid_the_facts).

[psader]

There's also #285 to resolve this.

[jakirkham]

Should this be closed now that PR ( #285 ) is merged or is this still outstanding?

[people-can-fly]

one workaround from @Ariel-Lieberman using java.long.string

Stack = autoclass('java.util.Stack')
jString = autoclass('java.lang.String')
stack = Stack()
stack.push(jString('OK'))
print(stack.pop())

[KeyWeeUsr]

Resolved by #285. If the issue still persists, open a new issue and reference old ones if they are related. This one is rather messy together with the references.