Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Deadlock while using wifi routines from the timer task (IDFGH-1143) #3458

Closed
Sermus opened this issue May 11, 2019 · 4 comments
Closed

Deadlock while using wifi routines from the timer task (IDFGH-1143) #3458

Sermus opened this issue May 11, 2019 · 4 comments

Comments

@Sermus
Copy link

Sermus commented May 11, 2019

Environment

  • Development Kit: ESP32-Wrover-Kit|Custom hardware
  • Kit version: v2
  • Module or chip used: ESP32-WROOM-32
  • IDF version (run git describe --tags to find it): 3.1.3
  • Build System: make
  • Compiler version (run xtensa-esp32-elf-gcc --version to find it): 1.22.0-80-g6c4433a
  • Operating System: macOS
  • Power Supply: USB

Problem Description

There is a deadlock. WiFi and timer tasks fall into a deadlock by cross-acquiring timer protection mutex and a wifi mutex.
The callstacks for the corresponding threads are:
Thread 8 (Thread 1073500392):
#0 0x4008f499 in xQueueGenericReceive (xQueue=0x3ffaea74, pvBuffer=0x0, xTicksToWait=4294967295, xJustPeeking=0) at .../esp-idf/components/freertos/queue.c:1592
#1 0x4008f4f4 in xQueueTakeMutexRecursive (xMutex=0x3ffaea74, xTicksToWait=4294967295) at .../esp-idf/components/freertos/queue.c:635
#2 0x400d20b5 in esp_timer_delete (timer=0x3ffce4ac) at .../esp-idf/components/esp32/esp_timer.c:166
#3 0x400fb42e in ets_timer_done (ptimer=0x3ffbb0d4) at .../esp-idf/components/esp32/ets_timer_legacy.c:97
#4 0x400fb1c0 in timer_done_wrapper (ptimer=0x3ffbb0d4) at .../esp-idf/components/esp32/esp_adapter.c:350
#5 0x4010d03d in ieee80211_mlme_connect_bss ()
#6 0x40113d47 in ?? ()
#7 0x40114213 in ?? ()
#8 0x40114c94 in cnx_csa_fn_process ()
#9 0x40112f8b in ?? ()
#10 0x40113101 in ieee80211_timer_do_process ()
#11 0x4008da6b in ppTask ()
#12 0x4008eb49 in vPortTaskWrapper (pxCode=0x4008d9ac , pvParameters=0x0) at .../esp-idf/components/freertos/port.c:143

Thread 7 (Thread 1073412812):
#0 0x4008f499 in xQueueGenericReceive (xQueue=0x3ffc5250, pvBuffer=0x0, xTicksToWait=4294967295, xJustPeeking=0) at .../esp-idf/components/freertos/queue.c:1592
#1 0x400fb278 in semphr_take_wrapper (semphr=0x3ffc5250, block_time_tick=4294967295) at .../esp-idf/components/esp32/esp_adapter.c:234
#2 0x40108538 in ieee80211_ioctl ()
#3 0x400fcca2 in esp_wifi_sta_get_ap_info ()
#4 0x400ebcc9 in network::update_network_info (arg=) at .../board/src/commons/net/network/network.cpp:304
#5 0x400d1fbf in timer_process_alarm (dispatch_method=) at .../esp-idf/components/esp32/esp_timer.c:290
#6 0x400d2052 in timer_task (arg=0x0) at .../esp-idf/components/esp32/esp_timer.c:318
#7 0x4008eb49 in vPortTaskWrapper (pxCode=0x400d2034 <timer_task>, pvParameters=0x0) at .../esp-idf/components/freertos/port.c:143

The deadlock is induced by esp_wifi_sta_get_ap_info being called from the timer thread which is done within a scope protected by mutex 0x3ffaea74 because this is done in timer callback context. While executing esp_wifi_sta_get_ap_info it obviously tries to acquire semaphore 0x3ffc5250 which is supposedly already taken by the WiFi thread. The latter one at this moment tries to capture mutex 0x3ffaea74 during the timer delete routine. Finita...

Steps to repropduce

They are obvious from the callstacks.

Code to reproduce this issue

Invoke esp_wifi_sta_get_ap_info (or another wifi-related stuff) in timer thread. Repro rate is not 100%, but the symptoms are clear from the callstacks and analysis above.
@github-actions github-actions bot changed the title Deadlock while using wifi routines from the timer task Deadlock while using wifi routines from the timer task (IDFGH-1143) May 11, 2019
@Sermus
Copy link
Author

Sermus commented May 18, 2019 •
edited
Loading

Different scenario, same root cause.
I'm trying to invoke esp_reboot from the timer task.

Thread 8 (Thread 1073412856):
#0 0x40088a9c in xQueueGenericReceive (xQueue=0x3ffc52e8, pvBuffer=0x0, xTicksToWait=4294967295, xJustPeeking=0) at ...esp32/esp-idf/components/freertos/queue.c:1592
#1 0x400f71b5 in semphr_take_wrapper (semphr=0x3ffc52e8, block_time_tick=4294967295) at ...esp32/esp-idf/components/esp32/esp_adapter.c:236
#2 0x40102f38 in ieee80211_ioctl ()
#3 0x400f8244 in esp_wifi_stop ()
#4 0x40081b55 in esp_restart () at ...esp32/esp-idf/components/esp32/system_api.c:255
......

Thread 9 (Thread 1073500544):
#0 0x40088a9c in xQueueGenericReceive (xQueue=0x3ffaeaa0, pvBuffer=0x0, xTicksToWait=4294967295, xJustPeeking=0) at ...esp32/esp-idf/components/freertos/queue.c:1592
#1 0x40088afc in xQueueTakeMutexRecursive (xMutex=0x3ffaeaa0, xTicksToWait=4294967295) at ...esp32/esp-idf/components/freertos/queue.c:635
#2 0x400d1d95 in esp_timer_delete (timer=0x3ffcd738) at ...esp32/esp-idf/components/esp32/esp_timer.c:166
#3 0x400f7280 in ets_timer_done (ptimer=0x3ffb3ed8) at ...esp32/esp-idf/components/esp32/ets_timer_legacy.c:97
#4 0x400f70a8 in timer_done_wrapper (ptimer=0x3ffb3ed8) at ...esp32/esp-idf/components/esp32/esp_adapter.c:350
#5 0x400fb9a8 in wifi_softap_stop ()
#6 0x40100f4b in ?? ()
#7 0x401020e4 in wifi_stop_process ()
#8 0x40102e3d in ieee80211_ioctl_process ()
#9 0x4008e0ff in ppTask ()

@liuzfesp
Copy link
Contributor

liuzfesp commented May 20, 2019 •
edited
Loading

@Sermus, thanks for reporting this bug. We already have solution for it, will merge it once it passes internal code review and testing.

@liuzfesp
Copy link
Contributor

This issue is fixed in commit: 197e0ae

@Alvin1Zhang
Copy link
Collaborator

@Sermus Thanks for reporting the issue, feel free to reopen the issue if it still exists. Thanks.

igrr added a commit that referenced this issue Jul 3, 2019
@igrr
esp_timer: handle esp_timer_delete in timer task
6b08e8b 
Closes #3458
trombik pushed a commit to trombik/esp-idf that referenced this issue Aug 9, 2019
@igrr @trombik
esp_timer: handle esp_timer_delete in timer task
dd4b6e6 
Closes espressif#3458
@igrr igrr mentioned this issue Sep 26, 2019
Closed
espressif-bot pushed a commit that referenced this issue Nov 19, 2019
@igrr @espressif-bot
esp_timer: handle esp_timer_delete in timer task
a179125 
Closes #3458
espressif-bot pushed a commit that referenced this issue Dec 15, 2019
@igrr @espressif-bot
esp_timer: handle esp_timer_delete in timer task
0100769 
Closes #3458
0xFEEDC0DE64 pushed a commit to 0xFEEDC0DE64/esp-idf that referenced this issue May 5, 2021
@davefiddes @me-no-dev
SPI: Fix discarded-qalifiers warning when compiling with all warnings (…
dac493f 
…espressif#3458)

* SPI: Fix discarded-qalifiers warning when compiling with all warnings

This fixes an error introduced with changeset b847f41 which
tightened the use of const for read-only data. The helper
funtion __transferBytes also requires the const qualifier on
outgoing data. Without this change a warning is displayed
when compiling with the Arduino IDE set to display "All"
compiler warnings.

Tests:
 - Build an ESP32 SPI sketch that uses static const data to send
   to an SPI device using the SPI.transferBytes() API

* SPI:Ensure all local functions are marked static

This audits all functions in the esp32-hal-xpi.c module and
ensures that any functions entirely local to the module are
marked as static.

Tests:
 - Build with Arduino set to show all warnings and ensure none
   are displayed

* SPI: Remove unused local __spiTranslate24 function

This removes the __spiTranslate24() function which is unused.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@Sermus @liuzfesp @Alvin1Zhang