added support for secure TLS connections #699

Legion2 · 2020-07-14T21:23:40Z

support TLS on ESP32 and ESP8266 close #101

TODO:

documentation
tests

1technophile · 2020-07-15T01:49:26Z

Thanks for the PR, if you need, I would be able to test it next week.

Legion2 · 2020-07-15T07:06:02Z

I already did manual tests, but I want it to be "tested" in the ci.

Legion2 · 2020-07-16T22:03:46Z

@1technophile should we keep the Test_config.h in sync with the User_config.h?

1technophile · 2020-07-18T14:18:14Z

@1technophile should we keep the Test_config.h in sync with the User_config.h?

We should remove it, while we don't test Arduino IDE with the CI we don't need it anymore

Legion2 · 2020-07-18T19:47:39Z

how we then "test"(compile) the TLS code in the CI pipeline?

1technophile · 2020-07-18T20:18:23Z

If you need Test_config.h to store a test certificate you can keep it.
Do you need other things to test this function?

Legion2 · 2020-07-18T20:20:46Z

For just compiling we don't need a real certificat, and for manual testing I use my own ca from my MQTT broker.

Legion2 · 2020-07-20T16:37:08Z

@1technophile Were in the docs should I put the documentation for the TLS MQTT connection? Is there a section for the MQTT broker connection setup, I did not found any.

1technophile · 2020-07-20T16:58:42Z

@Legion2 maybe we should add an advanced_configuration.md in the beginning of the upload section?

Legion2 · 2020-07-20T17:12:42Z

An advanced configuration section sounds good.

The tests fail on ESP32 because there is no more free Flash when compiling the esp32dev-all example:

RAM:   [==        ]  22.0% (used 72156 bytes from 327680 bytes)
Flash: [==========]  104.9% (used 2062180 bytes from 1966080 bytes)
Error: The program size (2062180 bytes) is greater than maximum allowed (1966080 bytes)

1technophile · 2020-07-20T17:55:04Z

I don't think it is a big issue, as [env:esp32dev-all] is more a test environment, maybe you should add a test env dedicated to the certificate testing?

I think also that we should differentiate test and production env and do not upload the binaries to github for the test ones.

Legion2 · 2020-07-23T11:33:15Z

Ok good idea to make special test environments, @1technophile which of the current env do you consider to be only test related?
We can give them a test- prefix so we can filter them before upload.

1technophile · 2020-07-23T11:49:33Z

Here they are:

esp32dev-all
nodemcuv2-all
atmega-all
test-manual-wifi

We should keep on extracting the libraries from these environments as it enables to have all the libraries in one zip file. But their binaries are in my point of view not necessary and can mislead the user.

Legion2 · 2020-07-23T17:47:09Z

why does the atmega-all env not have a .bin file uploaded?

1technophile · 2020-07-23T17:49:24Z

I have never uploaded a BIN file directly on an Arduino Mega neither searched how to do it.

Legion2 · 2020-07-23T17:51:59Z

oh ok

Legion2 · 2020-07-23T17:53:10Z

currently we use the name of the env for the name of the uploaded files. So if we add test- also the library zip will start with test-.

1technophile · 2020-07-24T11:11:35Z

Maybe we should do it the other way and don't publish the binaries that end with -all?

Legion2 · 2020-07-24T11:12:50Z

but then other test- binaries are still published

1technophile · 2020-07-24T11:16:07Z

Like that?

*-all we publish libraries and not binaries
*-test we don't publish
others we publish libraries and binaries

Legion2 · 2020-07-24T11:40:26Z

@1technophile I create a new pull request for this change

Legion2 · 2020-07-24T21:43:54Z

@1technophile how do I add a new page to the docs?

1technophile · 2020-07-24T21:51:44Z

You create a new .md page in the wished directory and you add the directory/page_name in the config.js for the docs (sidebar section)

on ESP32 and ESP8266 close #101

fixed some some isues in the documentation

1technophile

Took me sometime to setup the necessary environment to test but now I'm up and running.

Thanks for the work first, this is a waited feature!

Here is my test report:

Board	Modules or env	Result	Comments
barebone ESP32	no	OK
barebone ESP32	IR & RF	OK
m5stickc	esp32-m5stick-c-ble	-	CoreDump after BLE Scan begin
barebone ESP32	esp32dev-ble	-	CoreDump after BLE Scan begin
nodemcuv2	RF, BT, DHT, ONOFF, ADC, GPIOInput	-	W: failed, ssl error code=54
nodemcuv2	RF	-	W: failed, ssl error code=54

With the nodemcu v2 I'm using the same MQTT server and the same certificate as the working esp32 configuration.
here is the log extract on the ESP8266:

E: Failed connecting 1st time to mqtt, you should put TRIGGER_GPIO to LOW or erase the flash
W: Wifi Protocol changed to WIFI_11G: 2
W: ESP8266: Forcing to wifi 2
W: MQTT connection...
W: failure_number_mqtt: 30
W: failed, rc=-2
W: failed, ssl error code=54

here is the coredump extract:

************* WELCOME TO OpenMQTTGateway **************
{"mqtt_server":"secure.les.com","mqtt_port":"8883","mqtt_user":"chevaliers","mqtt_pass":"duzodiaque","mqtt_topic":"home/","gateway_name":"OpenMQTTGateway_ESP32_M5STICK_C_BLE_IR"}*WM: [3] allocating params bytes: 20
*WM: [2] Added Parameter: server
*WM: [2] Added Parameter: port
*WM: [2] Added Parameter: user
*WM: [2] Added Parameter: pass
*WM: [2] Added Parameter: name
*WM: [3] Updated _max_params: 10
*WM: [3] re-allocating params bytes: 40
*WM: [2] Added Parameter: topic
N: Attempting Wifi connection with saved AP: 0
{"mqtt_server":"secure.les.com","mqtt_port":"8883","mqtt_user":"chevaliers","mqtt_pass":"duzodiaque","mqtt_topic":"home/","gateway_name":"OpenMQTTGateway_ESP32_M5STICK_C_BLE_IR"}N: BLEinterval: 55555
N: Minrssi: -100
N: Low Power Mode: 0
N: Setup OpenMQTTGateway end
W: MQTT connection...
N: Connected to broker
N: Subject: /SYStoMQTT
N: Received json : {"uptime":3,"version":"version_tag","freemem":95092,"rssi":-48,"SSID":"monreseau","ip":"9.9.9.9","mac":"D8:DD:EE:57:HH:55","wifiprt":0,"lowpowermode":0,"interval":55555,"modules":"BTHADiscovery"}
N: Scan begin
Guru Meditation Error: Core  0 panic'ed (StoreProhibited). Exception was unhandled.
Core 0 register dump:
PC      : 0x4000c46c  PS      : 0x00060f30  A0      : 0x801b3ff0  A1      : 0x3fff7610
A2      : 0x00000000  A3      : 0x00000000  A4      : 0x00000130  A5      : 0x00000000
A6      : 0x3f426318  A7      : 0x00000013  A8      : 0x80085039  A9      : 0x3fff75b0
A10     : 0x00000000  A11     : 0x00001800  A12     : 0x00000000  A13     : 0x00000003
A14     : 0x00000001  A15     : 0x00000006  SAR     : 0x00000008  EXCCAUSE: 0x0000001d
EXCVADDR: 0x00000000  LBEG    : 0x4000c46c  LEND    : 0x4000c477  LCOUNT  : 0x00000012

The main differences I see between the coredumping configurations and the others are:

less memory available due to the size of the BLE library
use of the 2 cores, the BLE scan is done on a different core

This is not related to your work I think but maybe a side effect.

Legion2 · 2020-07-25T20:23:35Z

W: failed, ssl error code=54

mean "Certificate is expired or not yet valid." this can be due to NTP does not set the time correctly. Can you confirm that the time is set correctly, this maybe take a while and it should be eventually set correctly. The normal connection retry should then successfully connect with the correct time set.

1technophile · 2020-07-25T21:33:41Z

mean "Certificate is expired or not yet valid." this can be due to NTP does not set the time correctly. Can you confirm that the time is set correctly, this maybe take a while and it should be eventually set correctly. The normal connection retry should then successfully connect with the correct time set.

On target, I just uncommented # define NTP_SERVER "pool.ntp.org" and now the ESP8266 is connected by TLS.
For info it can take several attempts before connecting as you mentionned.

Sumup

Board	Modules or env	Result	Comments
barebone ESP32	no	OK
barebone ESP32	IR & RF	OK
m5stickc	esp32-m5stick-c-ble	-	CoreDump after BLE Scan begin
barebone ESP32	esp32dev-ble	-	CoreDump after BLE Scan begin
nodemcuv2	RF, BT, DHT, ONOFF, ADC, GPIOInput	OK	-
nodemcuv2	RF	OK	-

Should we add something on the troubleshooting doc about W: failed, ssl error code=54 ?

Legion2 · 2020-07-25T22:06:44Z

Should we add something on the troubleshooting doc about W: failed, ssl error code=54 ?

yes I will also add # define NTP_SERVER "pool.ntp.org" to the docs

Legion2 · 2020-07-25T22:07:25Z

can you decode the core dump so we know in which function the error happens

1technophile · 2020-07-25T22:16:23Z

can you decode the core dump so we know in which function the error happens

Never done that, but there is always a first time! Let's try

1technophile · 2020-07-26T16:40:22Z

Here is the decoding of the stack trace:

PC: 0x4000c46c
EXCVADDR: 0x00000000

Decoding stack results
0x401bf165: bta_sys_init at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/bt/bluedroid/bta/sys/bta_sys_main.c line 170
0x40194591: btu_task_start_up at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/bt/bluedroid/stack/btu/btu_task.c line 284
0x401945cb: btu_task_thread_handler at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/bt/bluedroid/stack/btu/btu_task.c line 226
0x4008eb71: vPortTaskWrapper at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/port.c line 143

1technophile · 2020-07-26T16:43:53Z

Seems that we are not alone on this:
nkolban/esp32-snippets#842

1technophile · 2020-07-26T19:57:52Z

As a complementary info, I also tried to change the partition type to board_build.partitions = huge_app.csv with:
esp32dev-ble

I'm getting:

Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
RAM:   [==        ]  20.7% (used 67924 bytes from 327680 bytes)
Flash: [======    ]  60.1% (used 1891856 bytes from 3145728 bytes)

And still the core dump when the scan begin:

N: Changing LOW POWER mode to: 0
N: Scan begin
Guru Meditation Error: Core  0 panic'ed (StoreProhibited). Exception was unhandled.
Core 0 register dump:
PC      : 0x4000c46c  PS      : 0x00060f30  A0      : 0x801b47f0  A1      : 0x3fff7830
A2      : 0x00000000  A3      : 0x00000000  A4      : 0x00000130  A5      : 0x00000000
A6      : 0x3f426428  A7      : 0x00000013  A8      : 0x80085039  A9      : 0x3fff77d0
A10     : 0x00000000  A11     : 0x00001800  A12     : 0x00000000  A13     : 0x00000003
A14     : 0x00000001  A15     : 0x00000006  SAR     : 0x00000008  EXCCAUSE: 0x0000001d
EXCVADDR: 0x00000000  LBEG    : 0x4000c46c  LEND    : 0x4000c477  LCOUNT  : 0x00000012

Backtrace: 0x4000c46c:0x3fff7830 0x401b47ed:0x3fff7840 0x40188b29:0x3fff7860 0x40188b63:0x3fff7880 0x4008eb71:0x3fff78b0

Legion2 · 2020-07-27T16:44:21Z

yeah, from what I read, both (TLS and BLE) allocate huge amount of memory and so they can't be used together. We would have to do many memory usage optimizations to get both working at the same time.

1technophile · 2020-07-27T18:57:40Z

Some ideas here, here and here also.

I'm going to do some tests this week by changing different parameters and having a closer memory followup.

In all the case ESP-IDF will support NimBLE; a lighter BLE stack compared to BlueDroid, so as a last option we may add a note in the documentation about the current incompatibility and wait for the NimBLE integration into Arduino ESP32.

1technophile · 2020-08-02T16:12:26Z

A better solution, I have found this library:
https://github.com/h2zero/NimBLE-Arduino
Which consumes less memory compared to Bluedroid, so I propose to merge this PR and we will integrate the lib further.

Legion2 · 2020-08-02T16:22:18Z

Ok

Legion2 added the enhancement label Jul 14, 2020

Legion2 self-assigned this Jul 14, 2020

Legion2 mentioned this pull request Jul 14, 2020

add ssl support to connect to mqtt broker #101

Closed

Legion2 force-pushed the tls branch from 0b7a915 to 9ea25e8 Compare July 20, 2020 16:17

Legion2 force-pushed the tls branch from d25d203 to 20833f7 Compare July 24, 2020 21:32

Legion2 added 2 commits July 25, 2020 11:38

added support for secure TLS connections

4b42f89

on ESP32 and ESP8266 close #101

added TLS test builds

7fa76d7

Legion2 force-pushed the tls branch from 20833f7 to 7fa76d7 Compare July 25, 2020 09:39

added documentation for TLS

c08be02

fixed some some isues in the documentation

Legion2 marked this pull request as ready for review July 25, 2020 12:52

Legion2 requested a review from 1technophile July 25, 2020 12:52

1technophile reviewed Jul 25, 2020

View reviewed changes

added documentation for error code=54

2a6a8a4

1technophile merged commit b5527d2 into development Aug 2, 2020

1technophile deleted the tls branch August 2, 2020 16:29

added support for secure TLS connections #699

added support for secure TLS connections #699

Conversation

Legion2 commented Jul 14, 2020 • edited Loading

1technophile commented Jul 15, 2020

Legion2 commented Jul 15, 2020

Legion2 commented Jul 16, 2020

1technophile commented Jul 18, 2020

Legion2 commented Jul 18, 2020

1technophile commented Jul 18, 2020

Legion2 commented Jul 18, 2020

Legion2 commented Jul 20, 2020

1technophile commented Jul 20, 2020

Legion2 commented Jul 20, 2020

1technophile commented Jul 20, 2020

Legion2 commented Jul 23, 2020

1technophile commented Jul 23, 2020

Legion2 commented Jul 23, 2020

1technophile commented Jul 23, 2020

Legion2 commented Jul 23, 2020

Legion2 commented Jul 23, 2020

1technophile commented Jul 24, 2020

Legion2 commented Jul 24, 2020

1technophile commented Jul 24, 2020 • edited Loading

Legion2 commented Jul 24, 2020

Legion2 commented Jul 24, 2020

1technophile commented Jul 24, 2020

1technophile left a comment • edited Loading

Choose a reason for hiding this comment

Legion2 commented Jul 25, 2020

1technophile commented Jul 25, 2020 • edited Loading

Legion2 commented Jul 25, 2020

Legion2 commented Jul 25, 2020

1technophile commented Jul 25, 2020

1technophile commented Jul 26, 2020

1technophile commented Jul 26, 2020 • edited Loading

1technophile commented Jul 26, 2020

Legion2 commented Jul 27, 2020

1technophile commented Jul 27, 2020 • edited Loading

1technophile commented Aug 2, 2020 • edited Loading

Legion2 commented Aug 2, 2020

Legion2 commented Jul 14, 2020 •

edited

Loading

1technophile commented Jul 24, 2020 •

edited

Loading

1technophile left a comment •

edited

Loading

1technophile commented Jul 25, 2020 •

edited

Loading

1technophile commented Jul 26, 2020 •

edited

Loading

1technophile commented Jul 27, 2020 •

edited

Loading

1technophile commented Aug 2, 2020 •

edited

Loading