Problem in websocket with DHT [WEBSOCKET] #9 error(1002): Invalid UTF-8 in text frame

[karmacoma92]

Before creating a new issue please check that you have:

• searched the existing issues (both open and closed)
• searched the wiki
• asked for help in the chat
• done the previous things again :)

Fulfilling this template will help developers and contributors to address the issue. Try to be as specific and extensive as possible. If the information provided is not enough the issue will likely be closed.

You can now remove this line and the above ones. Text in italic is meant to be replaced by your own words. If any of the sections below are not relevant to the issue (for instance, the screenshots) then you can delete them.

Bug description
[WEBSOCKET] #9 error(1002): Invalid UTF-8 in text frame

I'm testing now with 2.6.3.
I do not have set WIFI_...
erase.config/ERASE.CONFIG does not help, -DPIO_FRAMEWORK_ARDUINO_ESPRESSIF_SDK221 doesn't help either.
I could only change the password, no wifi, nothing.
Relays doesn't seem to have the issue but I haven't tested any more

Steps to reproduce
You have to run esptool.py erase_flash and then flash the latest build 1.14.2-dev afterwards, or
flash directly latest 1.14.2-dev build on top of previous version from 23/01/20

Expected behavior
This is the expected behavior
[184195] [WEBSOCKET] #7 connected, ip: 192.168.20.222, url: /ws
[184363] [WEBSOCKET] #7 error(1002): Invalid UTF-8 in text frameevi� � [WEBSOCKET] #7 error(1002): Invalid UTF-8 in text frameevi� � [WEBSOCKET] #7 error(1002): Invalid UTF-8 in text frameevi� � [WEBSOCKET] #7 error(1002): Invalid UTF-8 in text frameevi� � [WEBSOCKET] #7 error(1002): Invalid UTF-8 in text frameevi� � [WEBSOCKET] #7 error(1002): Invalid UTF-8 in text frameevi� � [WEBSOCKET] #7 error(1002): Invalid UTF-8 in text frameevi� � [WEBSOCKET] #7 error(1002)
[184396] [WEBSOCKET] #7 disconnected

Screenshots
N/A

Device information

[000175] [MAIN] ESPURNA 1.14.2.PA5-dev (200219)
[000175] [MAIN] [email protected]
[000176] [MAIN] http://tinkerman.cat

[000177] [MAIN] CPU chip ID: 0x8E1600
[000180] [MAIN] CPU frequency: 80 MHz
[000184] [MAIN] SDK version: 2.2.2-dev(38a443e)
[000188] [MAIN] Core version: 2.6.3
[000191] [MAIN] Core revision: 3d128e5c
[000194] [MAIN] Build time: 1582320145
[000198]
[000199] [MAIN] Flash chip ID: 0x146085
[000202] [MAIN] Flash speed: 40000000 Hz
[000206] [MAIN] Flash mode: DOUT
[000208]
[000210] [MAIN] Flash size (CHIP) : 1048576 bytes / 256 sectors ( 0 to 255)
[000217] [MAIN] Flash size (SDK) : 1048576 bytes / 256 sectors ( 0 to 255)
[000224] [MAIN] Reserved : 4096 bytes / 1 sectors ( 0 to 0)
[000231] [MAIN] Firmware size : 569392 bytes / 140 sectors ( 1 to 140)
[000238] [MAIN] Max OTA size : 450560 bytes / 110 sectors ( 141 to 250)
[000245] [MAIN] EEPROM size : 4096 bytes / 1 sectors ( 251 to 251)
[000253] [MAIN] Reserved : 16384 bytes / 4 sectors ( 252 to 255)
[000260]
[000261] [MAIN] EEPROM sectors: 251, 250
[000264] [MAIN] EEPROM current: 251
[000267]
[000268] [MAIN] EEPROM: 4096 bytes initially | 1288 bytes used (31%) | 2808 bytes free (68%)
[000277] [MAIN] Heap : 40984 bytes initially | 5280 bytes used (12%) | 35704 bytes free (87%)
[000285] [MAIN] Stack : 4096 bytes initially | 1232 bytes used (30%) | 2864 bytes free (69%)
[000293]
[000294] [MAIN] Boot version: 31
[000297] [MAIN] Boot mode: 1
[000300] [MAIN] Last reset reason: External System
[000304] [MAIN] Last reset info: Fatal exception:0 flag:6 (EXT_SYS_RST) epc1:0x00000000 epc2:0x00000000 epc3:0x00000000 excvaddr:0x00000000 depc:0x00000000
[000317]
[000319] [MAIN] Board: GENERIC_ESP01S_DHT11_10
[000322] [MAIN] Support: ALEXA API BROKER BUTTON DEBUG_SERIAL DEBUG_TELNET DEBUG_WEB DOMOTICZ HOMEASSISTANT LED MDNS_SERVER MQTT NTP SCHEDULER SENSOR TELNET TERMINAL THINGSPEAK WEB
[000338] [MAIN] OTA: ARDUINO ASYNCTCP WEB
[000342] [MAIN] Sensors: DHTXX
[000345] [MAIN] WebUI image: SENSOR
[000348]
[000349] [MAIN] Power: 2475 mV
[000351] [MAIN] WiFi Sleep Mode: MODEM
[000355]

---8<-------

• Arduino Core version
  Core version 2.6.3
• ESPurna version
  1.14.2-dev
• Flash mode
  esptool but curl upgrade works as well
• Device brand, model and version
  generic-esp01s-dht11-10
  Tools used
• Desktop operating system
  Ubuntu 19.10
• Browser & version
  Firefox 73.0 (64-bit)
• IDE & version
  PlatformIO Core 4.2 and PIO Home 3.1
• Compiler & version (if not embedded in IDE)

Additional context
Add any other context about the problem here.

firmware.zip

tcpdump.zip

[mcspr]

Thanks!

Looking at the last message, before the UTF-8 error:

{"pwrUnits":0,"eneUnits":0,"tmpUnits":0,"tmpCorrection":-9.5,"humCorrection":22,"snsRead":6,"snsReport":10,"snsSave":0,"magnitudesConfig":{"index":[0,0],"type":[1,2],
"units":[>>>"°C"<<<,"%"],
"description":["DHT11 @ GPIO2","DHT11 @ GPIO2"],"size":2}}

Can you try changing this unit to just "C"?

espurna/code/espurna/sensor.ino

Line 85 in 8f7d77b

PROGMEM const char magnitude_celsius[] = "°C";

[karmacoma92]

Hi,
you made it work !!
thanks!
Oscar

[mcspr]

Cool.
What I don't get is why .ino file gets corrupted like that when translated into .cpp. Can you check what locale and file code/espurna/sensor.ino output is on that Ubuntu system?

[karmacoma92]

Hi

here is the output

espurna$ file code/espurna/sensor.ino
code/espurna/sensor.ino: C source, UTF-8 Unicode text

espurna$ locale
LANG=es_ES.UTF-8
LANGUAGE=
LC_CTYPE="es_ES.UTF-8"
LC_NUMERIC=en_GB.UTF-8
LC_TIME=en_GB.UTF-8
LC_COLLATE="es_ES.UTF-8"
LC_MONETARY=en_GB.UTF-8
LC_MESSAGES="es_ES.UTF-8"
LC_PAPER=en_GB.UTF-8
LC_NAME=en_GB.UTF-8
LC_ADDRESS=en_GB.UTF-8
LC_TELEPHONE=en_GB.UTF-8
LC_MEASUREMENT=en_GB.UTF-8
LC_IDENTIFICATION=en_GB.UTF-8
LC_ALL=

as far as I understand there is no problem with that, am I wrong?
Thanks

[mcspr]

Just wondering. For some reason °C isn't stored properly in the firmware file. For example, these are .elf contents of the section storing PROGMEM strings:

Yours:

(this is xxd output)
000861e0: 6d00 b567 2f6d b300 6b57 6800 4a00 6b57  m..g/m..kWh.J.kW
000861f0: 0057 0056 0041 0068 5061 0025 00b0 4600  .W.V.A.hPa.%..F.
00086200: b043 0000 5dc3 2740 51c3 2740 48c3 2740  .C..].'@Q.'@H.'@

Mine (built generic env with -DDHT_SUPPORT=1)

00086c30: 7070 6d00 c2b5 672f 6dc2 b300 6b57 6800  ppm...g/m...kWh.
00086c40: 4a00 6b57 0057 0056 0041 0068 5061 0025  J.kW.W.V.A.hPa.%
00086c50: 00c2 b046 00c2 b043 0000 0000 b5cd 2740  ...F...C......'@

°C is stored as C2 B0 43 for me but as B0 43 for you. nm even shows the size difference of the magnitude_celcius in that 1 byte:

$ xtensa-lx106-elf-nm -S issue2616.elf | grep magnitude_cel
4027c200 00000003 t _ZL17magnitude_celsius
$ xtensa-lx106-elf-nm -S mytestfirmware.elf | grep magnitude_cel
4027cc55 00000004 t _ZL17magnitude_celsius

edit: Why I was asking about the locale, this is what happens when converting stuff with pre-unicode era encoding:

>>> b'\xb0\x43'.decode('latin1')
'°C'
>>> b'\xb0\x43'.decode('utf-8')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 0: invalid start byte
>>> b'\xc2\xb0\x43'.decode('utf-8')
'°C'

[mcspr]

IDE & version
PlatformIO Core 4.2 and PIO Home 3.1

Based on my previous comment about decoding / encoding being wrong, I tried installing PIO with python2 and check what exactly happens with .ino conversion:

$ ~/pio2/bin/pio run -e itead-sonoff-basic-dht
...
Converting espurna.ino
Unicode decode error has occurred, please remove invalid (non-ASCII or non-UTF8) characters from /home/max/Projects/espurna/code/espurna/espurna.ino.cpp file
^Z
[1]+  Stopped                 ~/pio2/bin/pio run -e itead-sonoff-basic-dht
$ file espurna/espurna.ino.cpp
espurna/espurna.ino.cpp: C source, ISO-8859 text

Same thing with python3 version:

$ file espurna/espurna.ino.cpp
espurna/espurna.ino.cpp: C source, UTF-8 Unicode text

---

Since python2 is EOL, I'm inclined to leave things as they are.

$ ~/pio2/bin/pip install -U
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.

[karmacoma92]

hi,

do you think there might be something wrong in my FW file?

Oscar

[mcspr]

Yes. To fix the issue completely you would need to change PIO installation from python 2 to python 3. There is a workaround though, but I'd rather incentivise py3 migration.

…

[karmacoma92]

The same happens with generic-esp01s-ds18b20-10-ota, I believe it does make sense since it is ºC...

I know I'm having the one installed with pip2
pip -V
pip 20.0.2 from /usr/local/lib/python2.7/dist-packages/pip (python 2.7)

how would you go from pip2 to pip3?
Thanks
Oscar

edit: add missing 'you'

[mcspr]

Use pip3 / python3 -mpip and follow https://docs.platformio.org/en/latest/installation.html? I would assume Ubuntu comes with recent py3 version already installed, pip3 might be missing but is probably in some additional apt-get'able package

[karmacoma92]

how about this:

`[268971] [MQTT] Received domoticz/out => {
"Battery" : 255,
"RSSI" : 12,
"description" : "",
"dtype" : "General",
"hwid" : "5",
"id" : "00000001",
"idx" : 63,
[268982] [DOMOTICZ] Error parsing data
[268991] [MQTT] Received domoticz/out =>
"name" : "Cloud Cover",
"nvalue" : 0,
"stype" : "Percentage",
"svalue1" : "47.00",
"unit" : 1
}

[268998] [DOMOTICZ] Error parsing data
[282142] [SENSOR] DHT22 @ GPIO14 - temperature: 21.6C
[282143] [SENSOR] DHT22 @ GPIO14 - humidity: 42%
[292409] [MQTT] Received domoticz/out => {
"Battery" : 255`

Could it be something related to previous problem?
thanks
Oscar

[mcspr]

Unlikely. Can you open a separate issue about that? But, based on what I see, Received domoticz/out shows 2 different parts of the JSON object, which were received in two different messages.
You can also check what some other MQTT client or do a tcpdump to see raw mqtt packets.

[karmacoma92]

Ok, I will do that
thanks once more
Oscar

[mcspr]

Closing via platformio/platformio-core#3393

[karmacoma92]

Hi,
I'm afraid it doesn't work for me, I must say I haven't got the time to upgrade python to version 3, is it having something to do with that?

Thanks
Oscar

[karmacoma92]

I'm afraid I must be doing something wrong

pio upgrade --dev
Please wait while upgrading PlatformIO ...
PlatformIO has been successfully upgraded to 4.2.2b1
Release notes: https://docs.platformio.org/en/latest/history.html
/PlatformIO/Projects/espurna/code$ pio --version
PlatformIO, version 4.2.1

Oscar

[karmacoma92]

Hi
I have fixed the issue was happening:

pio --version
PlatformIO, version 4.2.2b1

platformio --version
PlatformIO, version 4.2.2b1

Apparently now it's working, I'll keep on testing tomorrow
thanks!
Oscar