HTTPS failure with low free heap memory?

[fabianoriccardi]

Hardware:

Board: Lolin D32
Core Installation/update date: commit: 884e417 06/dec/2018
IDE name: Arduino IDE
Flash Frequency: 40Mhz
PSRAM enabled: no
Upload Speed: 921600
Computer OS: Windows 10

Description:

I was testing the HTTPS functionality and it works perfectly. Last week I've added the BLE library, but it cause the failure of HTTPS calls. Actually Bluetooth is just instantiated BLEDevice::init(""); and never used, but it seems enough to make fail https calls. I think that the problem is the huge amount of ram stole by BLE library, but I'm not so confident about that.

This is what I get on serial port and it represent the common behaviour, everything works fine (even if there is that socket error) , you can see the huge (normal) amount of free heap:

[HTTPS] sending POST to myUrlAddress ...
Free Heap: 169876
[D][HTTPClient.cpp:265] beginInternal(): host: myHostAddress port: 443 url: myUrl
[E][WiFiClient.cpp:236] setSocketOption(): 1006 : 9
[I][ssl_client.cpp:151] start_ssl_client(): WARNING: Use certificates for a more secure communication!
[D][HTTPClient.cpp:977] connect():  connected to myHostAddress:443
[D][HTTPClient.cpp:1102] handleHeaderResponse(): code: 200
[D][HTTPClient.cpp:1105] handleHeaderResponse(): size: 45
[HTTPS] POST... code: 200
[D][HTTPClient.cpp:1239] writeToStreamDataBlock(): connection closed or file end (written: 45).
[D][HTTPClient.cpp:358] disconnect(): tcp stop

===========
{"content":null,"message":null,"result":"OK"}
===========
[HTTPS] end
[D][HTTPClient.cpp:369] disconnect(): tcp is closed

This is what I get when BLE is initialized (NOTE low free heap quantity):

[HTTPS] sending POST to myUrlAddress ...
Free Heap: 44952
[D][HTTPClient.cpp:265] beginInternal(): host: myHostAddress port: 443 url: myUrl
[E][WiFiClient.cpp:236] setSocketOption(): 1006 : 9
[I][ssl_client.cpp:151] start_ssl_client(): WARNING: Use certificates for a more secure communication!
[E][ssl_client.cpp:33] handle_error(): SSL - Memory allocation failed
[E][ssl_client.cpp:35] handle_error(): MbedTLS message code: -32512
[E][WiFiClientSecure.cpp:118] connect(): start_ssl_client: -32512
[D][HTTPClient.cpp:973] connect(): failed connect to myHostAddress:443
[W][HTTPClient.cpp:1262] returnError(): error(-1): connection refused
[HTTPS] POST... failed, error: connection refused
[HTTPS] end
[D][HTTPClient.cpp:369] disconnect(): tcp is closed

My questions are:

• are there tricks to save RAM memory taken by ble library?
• how many bytes takes an HTTPS call? Does it depends on message size?

[chegewara]

MbedTLS message code: -32512
mbedtTLS cant start because lack of heap. You need about 70-ish kB free heap i think to even start with mbedTLS.

[fabianoriccardi]

Thank, actually I've tried to allocate memory to see this limit. It seem that it start to fails if free heap is about 100k.

the code in the loop is:

  HTTPClient http;
  byte* buf = (byte*)malloc(step);
  if(buf==NULL) Serial.println("Error");
  i+=step;
  Serial.print("[HTTP] begin...\n");
  Serial.println(ESP.getFreeHeap());
  // This call makes a HTTPS call!
  http.begin("myurl.com", ca); 
  Serial.println(ESP.getFreeHeap());
  Serial.print("[HTTP] GET...\n");
  int httpCode = http.GET();
  if(httpCode > 0) {
    Serial.printf("[HTTP] GET... code: %d\n", httpCode);
    if(httpCode == HTTP_CODE_OK) {
        String payload = http.getString();
        Serial.println(payload);
    }
  } else {
    Serial.printf("[HTTP] GET... failed, error: %s\n", http.errorToString(httpCode).c_str());
  }

  http.end();

The result is:

[HTTP] begin...
Free Heap: 104928
[HTTP] GET...
[HTTP] GET... code: 200
Hello World!

[HTTP] begin...
Free Heap: 94912
[HTTP] GET...
[HTTP] GET... failed, error: connection refused

I've repeated this test multiple times changing the memory allocated at each iteration: 5k/10k/20k were tested leading to the same result.

[czuvich]

I see the same issues with a WROVER module. See my referenced thread I posted in ESP32 forums. I have almost 4MB of heap available, but it says memory allocation failed. Any help is greatly appreciated.

https://www.esp32.com/viewtopic.php?t=8475

[chegewara]

Only because you have 4MB free heap it does not means that all can be used in all situations of for all components.
start_ssl_client: -32512 means that mbedTLS does not have enough memory to start, most likely you have +/-70kB internal heap free (or less) and most reported heap is from spiram.
You can ask @tobozo how to make use of spiram, he told me today he resolved problems with ble scanner by using spiram.

[tobozo]

@czuvich Depending on the SDK version ESP.getFreeHeap() will return values calculated differently and may deceive. Until this is stabilized, I'll keep using heap_caps_get_free_size(MALLOC_CAP_INTERNAL) instead which sticks to the historical behaviour.

@fabiuz7 using http.getString() multiple times in a loop will eventually consume all available heap, Arduino Strings are evil :D

I've had the same issues in my own project, and upon @chegewara's suggestion I moved all String() statements to const char* and could eventually use ps_malloc instead of malloc.

Consequently you'll have to use streaming with HTTP.

// get this outside your loop !
HTTPClient http;
http.setReuse(true);
// assuming 'step' is a power of 2 (e.g. 512, 1024, 2048)
byte* buff = (byte*)ps_malloc(step); // only malloc/ps_malloc during setup() !!
// (...)

// modify your loop as follows:
WiFiClient * stream = http.getStreamPtr();
// (...)
if(httpCode == HTTP_CODE_OK) {
  while(http.connected() && (len > 0 || len == -1)) {
    // get available data size
    size_t size = stream->available();
    if(size) {
      // read up to 'step' bytes
      int c = stream->readBytes(buff, ((size > sizeof(buff)) ? sizeof(buff) : size));
      // write it to SD, Serial or any other resource
      Serial.write(buff, c);
      //SD.write(buff, c);
      if(len-c >= 0) {
        len -= c;
      }
    }
  }
}

If you're using a SD Card there's also the possibility to SD-load certificate data directly to the psram: bonus in sketch size and maintainability as it unlocks the possibility to have the certificate stored in psram.

[czuvich]

Thanks for the helpful responses!

I am getting the error on startup of the chip. If I call BLEDevice::init() followed by the HTTPS call it reports that error. If I omit BLEDevice::init() it’s fine.

So, how could I get that working in Arduino (without omitting one or the other)? Is the error reported to be expected since it sounds like it forces internal allocation? I’m looking at
CONFIG_WIFI_LWIP_ALLOCATION_FROM_SPIRAM_FIRST

Perhaps? I assume to get that working I’d need to recompile the SDK libs right?

[tobozo]

@czuvich maybe BLEDevice::getScan()->stop() and BLEDevice::getScan()->clearResults() before calling HTTPS could help free some space?

Ongoing scans can eat up to 256 bytes per detected device so it can affect your heap differently depending on how long your scan is.

[chegewara]

@czuvich Could you try revert order, first https calls and then ble?

[czuvich]

@chegewara Switching order allows me to execute WiFi and then BLE scans; however, I can't run WiFi afterwards. So, here's what I'm seeing:

BLE then WiFi (fails)
WiFi then BLE (success)
WiFi then BLE then WiFi (fails)

Shouldn't mbed use spiram in the Arduino environment? If not, I'm wondering if there needs to be some other configuration. Here's the full log from the other thread.

12:59:27.190 -> [V][ssl_client.cpp:53] start_ssl_client(): Free heap before TLS 4182376
12:59:27.190 -> [V][ssl_client.cpp:55] start_ssl_client(): Starting socket
[V][ssl_client.cpp:88] start_ssl_client(): Seeding the random number generator
12:59:27.684 -> [V][ssl_client.cpp:97] start_ssl_client(): Setting up the SSL/TLS structure...
12:59:27.684 -> [I][ssl_client.cpp:121] start_ssl_client(): WARNING: Use certificates for a more secure communication!
12:59:27.684 -> [V][ssl_client.cpp:145] start_ssl_client(): Setting hostname for TLS session...
12:59:27.684 -> [E][ssl_client.cpp:33] handle_error(): SSL - Memory allocation failed
12:59:27.684 -> [E][ssl_client.cpp:35] handle_error(): MbedTLS message code: -32512
12:59:27.717 -> [E][WiFiClientSecure.cpp:109] connect(): start_ssl_client: -32512
12:59:27.717 -> [V][ssl_client.cpp:211] stop_ssl_socket(): Cleaning SSL connection.
12:59:27.717 -> [D][HTTPClient.cpp:846] connect(): failed connect to [my backend server]:443
12:59:27.717 -> [W][HTTPClient.cpp:1124] returnError(): error(-1): connection refused
12:59:27.717 -> [I][HTTPAdapter.cpp:135] sync(): Received HTTP response [-1].
12:59:27.750 -> [W][HTTPClient.cpp:1124] returnError(): error(-4): not connected

[tobozo]

@czuvich on the Wrover kit + SDK-rc1 I had to run psramInit() manually before being able to use it, which is odd as it's supposed to be automatically initialized.

[czuvich]

@tobozo Thanks for suggestion, but that doesn't seem to work either. I am currently running v3.2-dev-39-gaaf12390 with the WROVER DEV Kit 4.1.

I think I may open up my own github issue for this problem. I don't think it's anything on my end, but I'll try to create a repo. Just curious, are you using the WROVER DEV Kit for your project? Did you end up getting your own board designed with a WROVER chip?

I am currently using the WROVER Dev kit; however, I don't need all of the extra "stuff".

[tobozo]

@czuvich I have the Wrover Dev Kit v4.1 too (playing with this project) without doing simultaneous WiFi/BLE though.

Not sure what the v3.2 you mention is, I'm on the arduino-esp32-1.0.1-rc2 of Arduino IDE 1.8.7 and using the 1.22.0-80-g6c4433a-5.2.0 xtensa.

[czuvich]

Arduino 1.0.1? Either that's a typo or you may want to update :) I'm on Arduino IDE 1.8.7. The SDK version I printed out was fromesp_get_idf_version(). Are you using a custom build for the ESP SDK, or are you using the Boards Manager build?

Also, thank you for linking your project. I can't release the source for mine, but I'm building a BLE->WiFi gateway using the WROVER dev kit for now. I'm leveraging the SD Card and an external RTC (DS3231).

[tobozo]

@czuvich In my project I have a use case for downloading HTTPS content between two BLE scans, what should I do in my scan loop to replicate your issue ?

btw on my build the RTC module wants to run on core 1 or it dies miserably.

[czuvich]

@tobozo I setup 2 tasks. One task is for BLEScanning and the other task is for posting data to an HTTPS backend. For the repo, I was going to use only 1 task and call BLEDevice::init() in setup since I am getting an error without scanning. I am using the following base class for executing tasks:

https://github.com/nkolban/esp32-snippets/blob/master/cpp_utils/Task.h

So something like this in the setup() function:

BLEDevice::init();
HttpPostTask task2 = new HttpPostTask(); // inherits from Task.h, feel free to configure stack size

Then, inside of the HttpPostTask.run(void* data), I use the WiFi and HttpClient.post() methods to establish a WiFi connection and POST data. I was going to just post something to https://www.google.com just to set it up.

That should be enough to replicate the issue. I've tried different stack allocations, but it always fails if BLEDevice::init() is invoked first.

To work around the issue, I just "schedule" the HTTPS POST. If the program detects that it's time to run the POST, it resets the board, executes the POST first, and resets the board again. I keep a persistent variable that tells me the next time to execute the POST (hence the RTC).

[czuvich]

btw on my build the RTC module wants to run on core 1 or it dies miserably.

So if you try to call the RTCLib.now() (not sure which RTCLib you're using) function on core 0 it will fail?

[tobozo]

@czuvich yep RTCLib.now() fails on core 0, returns 165:165:65 unless it runs on core 1, I haven't figured out why

I wont be able to test BLE+TLS on Arduino until a couple of days, currently it won't compile because other libraries of my app already use too much space, I'll update this comment with the progress.

[stickbreaker]

check the task priority.

I found issues with task starvation because of interrupt priority inversion. A high priority task waiting for a semaphore held by a low priority task that never gets any cpu cycles.

Chuck.

[czuvich]

Ok... So I took out both BLE and WiFi out of task execution and put them in the loop(), and I'm able to execute WiFi after BLE once; however, it fails the second time with the memory allocation error.

I've configured my tasks as follows:
BLE task pinned to core 1 with a stack size of 25,000 and priority of 5.
WiFi task pinned to core 1 with a stack size of 20,000 and priority of 5.

Maybe I have my tasks configured incorrectly? I'm using the Task.h base class to execute the tasks from:
https://github.com/nkolban/esp32-snippets/blob/master/cpp_utils/Task.h

[chegewara]

2 things:

1. your stack size for both tasks is way too big, and more important:
2. i could not get rid of memory leak in very simple esp-idf ble example and i decided quit it and use regular esp-idf task instead, such a comfort now
   PS im not saying this class is introducing memory leak, just i could not instantiate it and then destroy without memory leak

[czuvich]

@chegewara Thank you for finding that!

i could not get rid of memory leak in very simple esp-idf ble example

What was leaking? I am not allocating a new Task(), but instead I have a global task in the sketch that I re-use.

[chegewara]

In my example the issue was when i am using this example to connect and disconnect multiple times. Every connect/disconnect procedure takes about 3kB memory and cant free it:
https://github.com/nkolban/esp32-snippets/blob/master/cpp_utils/tests/BLETests/SampleClient.cpp

[czuvich]

@chegewara The Task.h wrapper is not really doing much, so I'd be surprised if that's the cause of the leak. I'll try to re-work my tasks then to see if that's where one of the issues is at, but even taking out the tasks I still get memory allocation failed on a WROVER.

In theory, I shouldn't be running into an allocation issue on this chip after 2 executions.

[chegewara]

I thought exactly the same, its just a task wrapper. But as soon as i switched in this example to normal freeRTOS task there is no longer memory leaks.

[czuvich]

Got it. I think I may need to just wait until 1.0.1 is officially released. I'm running into several issues with the WROVER dev kit and BLE using the latest dev branch. I'll post findings as I move along.

[czuvich]

@chegewara FYI, I think the leak in the Task.cpp is in line 84 - Task::stop().
I'm not entirely sure why there's a copy of the handle being made (maybe concurrency issue?)... but the handle should probably be deleted not a copy of it.

void Task::stop() {
if (m_handle == nullptr) return;
::vTaskDelete(m_handle);
m_handle = nullptr;
} // stop

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This stale issue has been automatically closed. Thank you for your contributions.

[igor-d-n]

What's wrong with Esp32? Why in 8266 I can establish 2 https connections with only 40-50Kb free heap and at esp32 it's falling all with 400+Kb?! Firmware size in flash is about 2Mb - wtf? It's soo unoptimized or what? May be switching to it at work project is going to be the greatest failure(

[me-no-dev]

yet there are countless devices out there running on ESP32... maybe something you are doing wrong?