mec15xxevb_assy6853:arch.arm.arch.arm.no.multithreading failed to run

[Zhaoningx]

Describe the bug
It will make system crash, when running test cast of tests/arch/arm/arm_no_multithreading/

To Reproduce
Steps to reproduce the behavior:

west build -b mec15xxevb_assy6853 -p auto tests/arch/arm/arm_no_multithreading/
~/software/ci/tools/usbgpio_ftdi/pre_script.sh
west flash
~/software/ci/tools/usbgpio_ftdi/post_script.sh
see error

Logs and console output
*** Booting Zephyr OS build zephyr-v2.3.0-1774-g34859d0d3e40 ***
ARM no-multithreading test
Available IRQ line: 173
ASSERTION FAIL [flag > 0] @ WEST_TOPDIR/zephyr/tests/arch/arm/arm_no_multithreading/src/main.c:105
Test flag not set by IRQ

E: ***** HARD FAULT *****
E: Fault escalation (see below)
E: r0/a1: 0x00000004 r1/a2: 0x00000069 r2/a3: 0x00000000
E: r3/a4: 0x00000000 r12/ip: 0x00000000 r14/lr: 0x000e0733
E: xpsr: 0x61000000
E: Faulting instruction address (r15/pc): 0x000e4bec
E: >>> ZEPHYR FATAL ERROR 0: CPU exception on CPU 0
E: Current thread: 0x00000000 (Y�)
E: Halting system

Environment (please complete the following information):

OS: Fedora28
Toolchain: Zephyr-sdk-0.11.4
Commit ID: 34859d0d3e

[ioannisg]

Thanks for the bug report @Zhaoningx
I don't have this platform to reproduce this issue, but it seems to me that it might not be possible to trigger (pend and unpend) this particular ISR. In that case i assume we can exclude the platform from the test.

Could you please

• check if the other tests/arch/arm tests are executed properly on this platform? Please, add the log output, here, if possible
• check if the test fails on current master (we've recently modified the logic around selecting the IRQ line for the test)

[ioannisg]

Setting bug priority to LOW, for now. It seems the test passes on multiple cortex-m platforms. Waiting for more input from @Zhaoningx

[Zhaoningx]

Hi @ioannisg , I test two tests of the other tests/arch/arm and the log output as below:

1. tests/arch/arm/arm_interrupt/
*** Booting Zephyr OS build zephyr-v2.3.0-1774-g34859d0d3e40 ***
Running test suite arm_interrupt

===================================================================
START - test_arm_interrupt
Available IRQ line: 173
E: >>> ZEPHYR FATAL ERROR 1: Unhandled interrupt on CPU 0
E: Current thread: 0x00118890 (unknown)
Caught system error -- reason 1
E: r0/a1: 0x00000003 r1/a2: 0x00118708 r2/a3: 0x00000003
E: r3/a4: 0x00118090 r12/ip: 0x00000000 r14/lr: 0x000e181f
E: xpsr: 0x610000bd
E: Faulting instruction address (r15/pc): 0x000e064e
E: >>> ZEPHYR FATAL ERROR 3: Kernel oops on CPU 0
E: Fault during interrupt handling

E: Current thread: 0x00118890 (unknown)
Caught system error -- reason 3
E: r0/a1: 0x00000004 r1/a2: 0x00118708 r2/a3: 0x00000004
E: r3/a4: 0x00118090 r12/ip: 0x00000000 r14/lr: 0x000e181f
E: xpsr: 0x610000bd
E: Faulting instruction address (r15/pc): 0x000e0668
E: >>> ZEPHYR FATAL ERROR 4: Kernel panic on CPU 0
E: Fault during interrupt handling

E: Current thread: 0x00118890 (unknown)
Caught system error -- reason 4
ASSERTION FAIL [0] @ WEST_TOPDIR/zephyr/tests/arch/arm/arm_interrupt/src/arm_interrupt.c:49
Intentional assert

E: r0/a1: 0x00000004 r1/a2: 0x00000031 r2/a3: 0x00118890
E: r3/a4: 0x000e062d r12/ip: 0x00000000 r14/lr: 0x000e181f
E: xpsr: 0x610000bd
E: Faulting instruction address (r15/pc): 0x000e6156
E: >>> ZEPHYR FATAL ERROR 4: Kernel panic on CPU 0
E: Fault during interrupt handling

E: Current thread: 0x00118890 (unknown)
Caught system error -- reason 4
ASSERTION FAIL [0] @ WEST_TOPDIR/zephyr/drivers/timer/sys_clock_init.c:23
E: ***** HARD FAULT *****
E: Fault escalation (see below)
E: r0/a1: 0x00000004 r1/a2: 0x00000017 r2/a3: 0x00118890
E: r3/a4: 0x00000000 r12/ip: 0x00000000 r14/lr: 0xfffffff1
E: xpsr: 0x6100000f
E: Faulting instruction address (r15/pc): 0x000e6156
E: >>> ZEPHYR FATAL ERROR 0: CPU exception on CPU 0
E: Fault during interrupt handling

E: Current thread: 0x00118890 (unknown)
Caught system error -- reason 0
PASS - test_arm_interrupt

===================================================================
START - test_arm_user_interrupt
Skipped
PASS - test_arm_user_interrupt

===================================================================
Test suite arm_interrupt succeeded

===================================================================
PROJECT EXECUTION SUCCESSFUL

2.tests/arch/arm/arm_irq_advanced_features/
*** Booting Zephyr OS build zephyr-v2.3.0-1774-g34859d0d3e40 ***
Running test suite arm_irq_advanced_features

===================================================================
START - test_arm_dynamic_direct_interrupts
PASS - test_arm_dynamic_direct_interrupts

===================================================================
START - test_arm_zero_latency_irqs
Available IRQ line: 172
PASS - test_arm_zero_latency_irqs

===================================================================
Test suite arm_irq_advanced_features succeeded

===================================================================
PROJECT EXECUTION SUCCESSFUL

3.tests/arch/arm/arm_no_multithreading/ ran report at the lasting master
*** Booting Zephyr OS build zephyr-v2.3.0-1883-g4bd247af700a ***
ARM no-multithreading test
Available IRQ line: 173
ASSERTION FAIL [flag > 0] @ WEST_TOPDIR/zephyr/tests/arch/arm/arm_no_multithreading/src/main.c:117
Test flag not set by IRQ

E: ***** HARD FAULT *****
E: Fault escalation (see below)
E: r0/a1: 0x00000004 r1/a2: 0x00000075 r2/a3: 0x00000000
E: r3/a4: 0x00000000 r12/ip: 0x00000000 r14/lr: 0x000e071f
E: xpsr: 0x61000000
E: Faulting instruction address (r15/pc): 0x000e4be0
E: >>> ZEPHYR FATAL ERROR 0: CPU exception on CPU 0
E: Current thread: 0x00000000 (Y�)
E: Halting system

[ioannisg]

Thanks a lot for the log @Zhaoningx
I wonder if there's is a problem with the NVIC controller on this board.
In the failing test: could you try to un-pend the interrupt and pend it again after it is enabled?
So replace:
NVIC_EnableIRQ(i);

with

NVIC_ClearPendingIRQ(i);
NVIC_EnableIRQ(i);
NVIC_SetPendingIRQ(i);

and see if you can get the IRQ triggered?

[Zhaoningx]

Hi @ioannisg , I replaced this at the end of the file /tests/arch/arm/arm_no_multithreading/src/main.c:110. But the report log is same as the failure log before, also failure.

[ioannisg]

So is it a problem that the IRQ is not triggered at all? Or that the test_flag is not updated?

[Zhaoningx]

It showed me that ASSERTION FAIL [flag > 0] @ WEST_TOPDIR/zephyr/tests/arch/arm/arm_no_multithreading/src/main.c:119 Test flag not set by IRQ.

[ioannisg]

It showed me that ASSERTION FAIL [flag > 0] @ WEST_TOPDIR/zephyr/tests/arch/arm/arm_no_multithreading/src/main.c:119 Test flag not set by IRQ.

Yeap, I understand that. I wonder if you could verify whether the ISR handler has run (and updated the test_flag) or not. Maybe with some debugger, or just a printf you could check this :)

[Zhaoningx]

Hi @ioannisg , yup, I debugged it followed your idea, I added 'printk' into the ISR handler function, and hand and tail of main function, it showed me that the handler function didn't run, and the test_flag didn't updated, either, stay zero all time.
The test output log as below:
*** Booting Zephyr OS build zephyr-v2.3.0-1774-g34859d0d3e40 ***
ARM no-multithreading test
main first test_flag is 0
Available IRQ line: 173
main end test_flag = 0
ASSERTION FAIL [flag > 0] @ WEST_TOPDIR/zephyr/tests/arch/arm/arm_no_multithreading/src/main.c:108
Test flag not set by IRQ

E: ***** HARD FAULT *****
E: Fault escalation (see below)
E: r0/a1: 0x00000004 r1/a2: 0x0000006c r2/a3: 0x00000000
E: r3/a4: 0x00000000 r12/ip: 0x00000000 r14/lr: 0x000e075b
E: xpsr: 0x61000000
E: Faulting instruction address (r15/pc): 0x000e4c18
E: >>> ZEPHYR FATAL ERROR 0: CPU exception on CPU 0
E: Current thread: 0x00000000 (Y�)
E: Halting system

As you see, there isn't output inform print in the arm_isr_handler function.

[ioannisg]

@Zhaoningx thanks, so it looks like something blocks the ISR to run.
From what i see

• the IRQs are unconditionally enabled in line 59 (arch_irq_unlock(0);)
• the loop determines an NVIC line which is, available, disabled, and triggered (pended)
• the arch_irq_connect_dynamic installs the handler
• the NVIC_EnableIRQ(i) enables the interrupt, so it should be triggered

and you said you replaced NVIC_EnableIRQ(i) with
NVIC_ClearPendingIRQ(i);
NVIC_EnableIRQ(i);
NVIC_SetPendingIRQ(i);

Unless I am missing something, it seems weird that the ISR line is not triggered.
Can you check that CONFIG_DYNAMIC_INTERRUPTS is enabled?

Also, right before the ASSERT line can you add a check to see if the IRQs are enabled?

[Zhaoningx]

Hey @ioannisg , the macro CONFIG_DYNAMIC_INTERRUPTS=y has done in prj.conf. And I asserted the IRQs i, it showed me the output as follows:

*** Booting Zephyr OS build zephyr-v2.3.0-1774-g34859d0d3e40 ***
ARM no-multithreading test
main first test_flag is 0
Available IRQ line: 173
main end test_flag = 0
ASSERTION FAIL [arch_irq_is_enabled(i) == true] @ WEST_TOPDIR/zephyr/tests/arch/arm/arm_no_multithreading/src/main.c:108
Assert failed

E: ***** HARD FAULT *****
E: Fault escalation (see below)
E: r0/a1: 0x00000004 r1/a2: 0x0000006c r2/a3: 0x00000000
E: r3/a4: 0x00000000 r12/ip: 0x00000000 r14/lr: 0x000e0761
E: xpsr: 0x61000000
E: Faulting instruction address (r15/pc): 0x000e4c44
E: >>> ZEPHYR FATAL ERROR 0: CPU exception on CPU 0
E: Current thread: 0x00000000 (Y�)
E: Halting system

[ioannisg]

OK then this means you can pend an interrupt but not enable it.
Can you try to start the detection logic from a different starting point in the for loop:
for (i = CONFIG_NUM_IRQS - 1; i >= 0; i--) {

E.g. start from i = CONFIG_NUM_IRQS/2, so the CPU will find another NVIC line.
See if the test could pass in this case.

[Zhaoningx]

Hi @ioannisg , I have a try to preset some starting points to test this like /2 /3 /6 /10 and so on, but it also returns a error 'ASSERT failed to indicate that IRQs enabled failed', the reports as below:
*** Booting Zephyr OS build zephyr-v2.3.0-1774-g34859d0d3e40 ***
ARM no-multithreading test
main first test_flag is 0
Available IRQ line: 86
main end test_flag = 0
ASSERTION FAIL [arch_irq_is_enabled(i) == 1] @ WEST_TOPDIR/zephyr/tests/arch/arm/arm_no_multithreading/src/main.c:108
Assert failed

E: ***** HARD FAULT *****
E: Fault escalation (see below)
E: r0/a1: 0x00000004 r1/a2: 0x0000006c r2/a3: 0x00000000
E: r3/a4: 0x00000000 r12/ip: 0x00000000 r14/lr: 0x000e0761
E: xpsr: 0x61000000
E: Faulting instruction address (r15/pc): 0x000e4c44
E: >>> ZEPHYR FATAL ERROR 0: CPU exception on CPU 0
E: Current thread: 0x00000000 (Y�)
E: Halting system
*** Booting Zephyr OS build zephyr-v2.3.0-1774-g34859d0d3e40 ***
ARM no-multithreading test
main first test_flag is 0
Available IRQ line: 57
main end test_flag = 0
ASSERTION FAIL [arch_irq_is_enabled(i) == 1] @ WEST_TOPDIR/zephyr/tests/arch/arm/arm_no_multithreading/src/main.c:108
Assert failed

E: ***** HARD FAULT *****
E: Fault escalation (see below)
E: r0/a1: 0x00000004 r1/a2: 0x0000006c r2/a3: 0x00000000
E: r3/a4: 0x00000000 r12/ip: 0x00000000 r14/lr: 0x000e0761
E: xpsr: 0x61000000
E: Faulting instruction address (r15/pc): 0x000e4c44
E: >>> ZEPHYR FATAL ERROR 0: CPU exception on CPU 0
E: Current thread: 0x00000000 (Y�)
E: Halting system
*** Booting Zephyr OS build zephyr-v2.3.0-1774-g34859d0d3e40 ***
ARM no-multithreading test
main first test_flag is 0
Available IRQ line: 28
main end test_flag = 0
ASSERTION FAIL [arch_irq_is_enabled(i) == 1] @ WEST_TOPDIR/zephyr/tests/arch/arm/arm_no_multithreading/src/main.c:108
Assert failed

E: ***** HARD FAULT *****
E: Fault escalation (see below)
E: r0/a1: 0x00000004 r1/a2: 0x0000006c r2/a3: 0x00000000
E: r3/a4: 0x00000000 r12/ip: 0x00000000 r14/lr: 0x000e0745
E: xpsr: 0x61000000
E: Faulting instruction address (r15/pc): 0x000e4c2c
E: >>> ZEPHYR FATAL ERROR 0: CPU exception on CPU 0
E: Current thread: 0x00000000 (Y�)
E: Halting system
*** Booting Zephyr OS build zephyr-v2.3.0-1774-g34859d0d3e40 ***
ARM no-multithreading test
main first test_flag is 0
Available IRQ line: 16
main end test_flag = 0
ASSERTION FAIL [arch_irq_is_enabled(i) == 1] @ WEST_TOPDIR/zephyr/tests/arch/arm/arm_no_multithreading/src/main.c:108
Assert failed

E: ***** HARD FAULT *****
E: Fault escalation (see below)
E: r0/a1: 0x00000004 r1/a2: 0x0000006c r2/a3: 0x00000000
E: r3/a4: 0x00000000 r12/ip: 0x00000000 r14/lr: 0x000e0745
E: xpsr: 0x61000000
E: Faulting instruction address (r15/pc): 0x000e4c2c
E: >>> ZEPHYR FATAL ERROR 0: CPU exception on CPU 0
E: Current thread: 0x00000000 (Y�)
E: Halting system
*** Booting Zephyr OS build zephyr-v2.3.0-1774-g34859d0d3e40 ***
ARM no-multithreading test
main first test_flag is 0
Available IRQ line: 173
main end test_flag = 0
ASSERTION FAIL [arch_irq_is_enabled(i) == 1] @ WEST_TOPDIR/zephyr/tests/arch/arm/arm_no_multithreading/src/main.c:108
Assert failed

E: ***** HARD FAULT *****
E: Fault escalation (see below)
E: r0/a1: 0x00000004 r1/a2: 0x0000006c r2/a3: 0x00000000
E: r3/a4: 0x00000000 r12/ip: 0x00000000 r14/lr: 0x000e0763
E: xpsr: 0x61000000
E: Faulting instruction address (r15/pc): 0x000e4c44
E: >>> ZEPHYR FATAL ERROR 0: CPU exception on CPU 0
E: Current thread: 0x00000000 (Y�)
E: Halting system

Waiting for your replying.

[carlescufi]

@Zhaoningx you need to disable all drivers or subsystems that make use of kernel blocking calls such as mutexes or semaphores.

[Zhaoningx]

Hi @carlescufi , I have a try to disable all device drivers that the board mec15xxevb_assy6853 supported, and extra configure as below:
CONFIG_GPIO=n
CONFIG_I2C=n
CONFIG_ADC=n
CONFIG_USB=n
CONFIG_COUNTER=n
CONFIG_ESPI=n
CONFIG_PWM=n
CONFIG_WATCHDOG=n
CONFIG_PS2=n
CONFIG_PECI=n
CONFIG_KSCAN=n

but the bug is still met, so I think we should re-open this bug to continue to work on it. :)

[maksimmasalski]

@Zhaoningx I don't see error now in the latest main 1921a2d
Close the issue?

*** Booting Zephyr OS build zephyr-v2.4.0-654-g1921a2d84ecb  ***
ARM no-multithreading test
Available IRQ line: 173
ARM no multithreading test successful
===================================================================
PROJECT EXECUTION SUCCESSFUL

[Zhaoningx]

This issue has solved, so close it.