WiFi connected, have IP, but IP stack broken (IDFGH-9) #2184
Comments
|
I just reproduced this again after about 15 disconnect / reconnect cycles, and once again I saw the following messages right when it failed: |
FayeY
changed the title
EnvironmentDevelopment Kit: WROVER-KIT-V3 We seem to be experiencing the same issue. Although ours shows when I make a number of requests to the local HTTP server running on the device or making a HTTP request while AWS IoT is connecting, not after WiFi disconnect / reconnect. NB: I have just had a couple of instances where this error does not stop the TCP/IP stack from working. The error shows in the console output, but networking continues to work. It seems random at the moment. I am reviewing my HTTP server code in detail now to try rule out the possibility of this causing the @vonnieda - is your |
|
HI @talss89, no I'm not using AWS IOT, but I am using a custom port of Apache Paho and they are similar, and both use mbedtls. For what it's worth, I see this error both when connecting to MQTT and when trying to HTTPS connections, and as you note, it does seem to happen as soon as I start to connect to a service, rather than as soon as WiFi connects. I have long suspected this had something to do with mbedtls, but it's only a guess. |
|
Thanks @vonnieda, that's really useful info. And seems plausible. When I get a moment, I'll have a play with some of the mbedtls kconfig parameters. I wonder if disabling hardware AES has any effect... I'll update this issue if I figure anything out. |
|
Alright, I think I am pretty close to a root cause on this. It appears to be related to memory allocation in the WiFi or TCP stack. I managed to get a new error message that seems to back this up: When my app starts it immediately tries to connect to a stored WiFi AP. Another task is blocking, waiting for WiFi to be connected, and as soon as it is, it tries to connect to MQTT over TLS. And yet another task is trying to send some requests to an HTTPS API if WiFi is connected. I think what is actually happening is that DMA capable RAM is being exhausted. If I put a long delay in my code before hitting the HTTPS API everything is fine until it tries to connect, then I get the "null" failure. Here is a log with some memory logging included: I'm investigating now if I can find a way to free up more DMA capable RAM to see if that helps. One other note: It seems like when these TLS connections fail they don't give back the RAM. Could be a memory leak, or could just be that this is an unrecoverable error and I should be rebooting anyway. And finally, I'll note that I'm now on the 3.0.2 release instead of the detached HEAD I was using before. Thanks, |
|
Okay, I've confirmed that the above is true. I was able to replace the mbedtls allocator with my own by uncommenting my_calloc and my_free are just functions that use SPI RAM instead of internal RAM. So,now the question is how do I override the built in MBEDTLS_PLATFORM_MEMORY without having to hack at the IDF installation. I suppose I can make a local copy of the mbedtls component but it would be nice if this is define were configurable from menuconfig. And FWIW, I just ran another test where instead of adding the custom allocator I decreased CONFIG_SPIRAM_MALLOC_ALWAYSINTERNAL in menuconfig to 256 and that seems to do the trick as well. So, I think the main issue here is that if you are low on internal memory (below about 60k), mbedtls might eat it all, leaving none for the WiFi or TCP stack. Jason |
|
Hi @vonnieda , I replied to your post on the forums about this issue, here: |
|
HI @vonnieda, could you check your menuconfig: components -> ESP32-specific -> SPI RAM config -> Try to allocate memories of WiFi and LWIP in SPIRAM firstly. If failed, allocate internal memory. If it's not enabled, enabled it and then to check whether it can solve your issue. |
|
Hi @liuzfesp - yes, I have this option enabled. It does not solve the issue. The issue, as noted in the forum, is simply that there is no enough internal memory for multiple TLS connections to be open at the same time. I have about 89k in FreeRTOS stacks, along with a number of fixed DMA buffers, and WiFi and BLE, and this leaves me with too little internal memory. I think we can probably close this issue, since at it's core it's just an out of memory error. Perhaps it would be nice if the wifi "null" error message with a bit more descriptive. |
|
Well it should probably at least generate a system event so the app can do something about it. |
|
We also had "Try to allocate memories of WiFi and LWIP in SPIRAM firstly" set. I like the idea of generating a system event on wifi alloc fail. Thanks all. |
|
We already have a feature which is used to optimize WiFi log, not only the log printed when not enough internal memory, but also the other WiFi log. Currently this feature has a related low priority compare to others. Will update to you once the log feature is completed. As for the system event, I'm not sure whether it can be successfully post to the event task when there is not enough memory. |
|
Moreover, WiFi stack only print log when failed to allocate the management buffer. For data buffer, it's normal if the memory is not enough in some scenario, for example, test throughput with iperf etc. |
|
I am getting similar error I am using WIFI with external RAM. Internal Heap free: 41120 Plenty of external ram and some internal RAM. Does in default settings WIFI only uses internal ram. |
|
Yes, @snahmad, the default setting is interval RAM. |
|
I am using latest ESP-IDF SDK 3.2. Is there issue using latest not stable version. |
|
Is this issue resolve. |
|
CONFIG_WIFI_LWIP_ALLOCATION_FROM_SPIRAM_FIRST=y make it work. |
projectgus
changed the title
|
Hi @snahmad if SPIRAM is enabled, CONFIG_WIFI_LWIP_ALLOCATION_FROM_SPIRAM_FIRST should be enabled, otherwise you will encounter memory issue. |
|
Hi @liuzfesp, Is it compulsory to enable CONFIG_WIFI_LWIP_ALLOCATION_FROM_SPIRAM_FIRST when SPI PSRAM is enabked and wifi is also used in parallel? If that is compulsory then is there any meaning to give configurable option into menuconfig? Please clear out it so that we can use PSRAM along with WiFi together. |
|
Hi @liuzfesp and @Ritesh236, I have enabled this option but still getting issue. I am using esp-idf 3.2 stable version. is there some another settings? |
|
Hello ESP32 SDK Developers, Would you please help to bruklesh regarding issue which he is facing while using wifi with PARAM? |
|
Hi @burkulesomesh43 & @Ritesh236, My guess is that this isn't a bug in ESP-IDF, and is probably related to your application code. I am not a contributor, but this might help:
Read these pages, and it should become clear:
Although the above docs are for 'latest', a lot of the fundamentals apply to ESP-IDF v3. liuzfesp did a good write-up on Finally, it may be better to post on the ESP32 forums, rather than this issue, as AFAIK it's not a bug in IDF. I hope you manage to fix your problem. Good luck. |
|
Thanks @talss89 for your valuable feedback. So, I have already checked few configurations earlier but still we will check it again and get back to you if any query regarding that. And also, We will create issue over ESP32 community with details for same to get any other help regarding same. Thanks again for your valuable feedback Regards, |
|
@talss89 Thank you for your response. Regards, |
Hi talss89, |
|
I have tried all of the above but I still have the issue. can we please re open it? |
Jason, I'm having a similar issue with my ESP32-WROVER and want to try what you suggest above, but am not quite sure how to achieve it. I see from another post that MBEDTLS_PLATFORM_MEMORY is now added as a config option, so I believe I should set CONFIG_MBEDTLS_PLATFORM_MEMORY=y (I'm using mongoose-os), but I'm not clear how or where to implement |
|
Hi @gadget-man, You should call The function pointers you pass just need to mirror the calloc and free functions. Mine are complicated, because I use a custom allocator, but it could be as simple as something like this: |
Many thanks for the quick response. When I call |
|
You may need to #include "mbedtls/platform.h"
…On Tue, Jul 21, 2020 at 9:54 AM gadget-man ***@***.***> wrote:
Hi @gadget-man <https://github.com/gadget-man>,
You should call mbedtls_platform_set_calloc_free as early in your bootup
code as you can. I'm not familiar with Mongoose, so I'm not sure if it does
stuff before handing off control, but in my case I call it at the very
beginning of my app_main() function.
The function pointers you pass just need to mirror the calloc
<http://www.cplusplus.com/reference/cstdlib/calloc/> and free
<http://www.cplusplus.com/reference/cstdlib/free/> functions. Mine are
complicated, because I use a custom allocator, but it could be as simple as
something like this:
static void* my_calloc(size_t num, size_t size) {
/**
* 1. Allocate num * size bytes from SPIRAM.
* 2. Check if the allocation succeeded. If it failed, return NULL.
* 3. Zero the allocated memory to adhere to the contract of calloc.
* 4. Return the pointer to the allocated memory.
*/
void* ptr = heap_caps_malloc(num * size, MALLOC_CAP_SPIRAM);
if (ptr == NULL) {
return NULL;
}
memset(ptr, 0, num * size);
return ptr;
}
static void my_free(void* ptr) {
/**
* Memory allocated with heap_caps_malloc() can be freed using the
* system free, but this is here as a demonstration as to how
* you'd override it if needed.
*/
free(ptr);
}
void app_main()
{
mbedtls_platform_set_calloc_free(my_calloc, my_free);
}
Many thanks for the quick response. When I call
mbdtls_platform_set_calloc_free in my app_init (mgos), I get an error on
compiling: implicit declaration of function
'mbedtls_platform_set_calloc_free'. I'll ask the mongoose guys what I'm
doing wrong.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#2184 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAJAU4YOZJXS7FWF2YFWNH3R4WT2PANCNFSM4FJPGHHA>
.
|
|
Unfortunately that doesn't work. I've also checked that config.h has been modified to include: |
I'm experiencing an issue where WiFi gets into a state where it thinks it's connected, and has an IP, but all communications to and from the ESP fail, including pings.
This seems to happen after some random number of disconnect / reconnect cycles. I will get a SYSTEM_EVENT_STA_CONNECTED followed by SYSTEM_EVENT_STA_GOT_IP but the ESP can't talk or be talked to.
Here is a log from a recent example:
I suspect that the following lines have something to do with it:
Once it gets in this state, it seems to persist until I force a WiFi disconnect and reconnect. During this time, attempting to ping the ESP from another computer on the network times out.
Environment
Thanks,
Jason
The text was updated successfully, but these errors were encountered: