WATCHDOG TIMEOUT - ZS HW 1.1, FW 23.05.04-release

[vom513]

Hello all,

First time I've seen this one. This has a Micro Center 64gb micro SD card in it. Generally stable in this machine (Sun SS20). Only one image, term. disabled via DIP switch. Nothing else on the SCSI bus. Only ini config is to enable debug. Pretty simple. Full log attached.

Here is a snippet of the errors:

[16371ms] WATCHDOG TIMEOUT at PC 0x080138E6 LR 0x00000008 attempting bus reset
[31387ms] WATCHDOG TIMEOUT at PC 0x08013872 LR 0x00000008 attempting bus reset
[46451ms] WATCHDOG TIMEOUT at PC 0x080138E6 LR 0x00000008 attempting bus reset
[61466ms] WATCHDOG TIMEOUT at PC 0x080138E6 LR 0x00000008 attempting bus reset
[76482ms] WATCHDOG TIMEOUT at PC 0x080138E6 LR 0x00000008 attempting bus reset
[91497ms] WATCHDOG TIMEOUT at PC 0x080138E6 LR 0x00000008 attempting bus reset

zululog.txt

[aperezbios]

Thanks for the report, @vom513. In order for us to be able to investigate it at all, we'll need a bit more detail. Please provide the whole log file, and preferably a log file with debug mode enabled., which will enable us to see what was happening just prior to the watchdog timeout. Debug mode must be enabled before the device powers on, if you're using the DIP switch. You can also set debug=1 in your zuluscsi.ini file, if you have one. Make sure the log file says debug mode is enabled.

[PetteriAimonen]

From the non-debug zululog.txt attached in the question, timeout is happening while receiving SCSI data from host system. Often this can be caused by noise or signal integrity issues, leading one end to receive too many or few pulses.

You mentioned that debug should be enabled via zuluscsi.ini. For some reason that does not appear to be taking effect.

Log indicates termination disabled, like you said. Is there a separate terminator at the end of the cable? Both ends of the cable should be terminated, one is usually on the host system motherboard and other in the last device.

[vom513]

I may have been misunderstanding the debug setting. My zuluscsi.ini file was:

[SCSI 3]
Debug = 1

Does Debug actually need set to 0 ? Even though I only have this for ID 3 - that's my only device normally. So wouldn't this have the same effect as putting the Debug out in the global section of ini ?

As for the termination, I have since put an external terminator on the back port of the machine. I'm going to let it run like this for a bit and see what happens. Of course something as spurious as this is rather hard to reproduce.

Thanks for taking a look at this. I will update this discussion if/when I have some more/better data.

[PetteriAimonen]

The section should be just [SCSI], the debug setting applies to all devices.

[vom513]

Understood. So does 0 or 1 enable it ?

[aperezbios]

@vom513 Enabled is 1. Disabled is 0, which is the default.

[vom513]

So this just happened on a totally different machine as well. Diff ZS board (well still a 1.1 with the same FW version - I just meant I have multiple ZS boards...)

I was doing a lot of read operations at the time (find / -type f | xargs cksum -a SHA512) Only writes should have been whatever background writes from the OS.

Of course I didn't have debug on this machine at the time :( I have since flipped the debug DIP switch. I will try to recreate this issue.

EDIT: I wanted to add: I have termination disabled on the ZS, and I use an external terminator on the SCSI port on the back of the machines. The SCSI cable in this most recent one that had this error is actually shielded with metal braid, and I even nicely covered the exposed wires at each end with copper tape. It should be pretty well protected from EMI.

[aperezbios]

@vom513 it would be helpful to know which SD card you were using (specifics are good, manufacturer, full model number if it's laser-engraved on the bottom of the SD card, etc. as it may assist us in reproducing. Even a non-debug log is better than nothing, especially if the firmware actually crashed, and wrote a zuluerr.txt file.

[vom513]

Good point. I'm using these (Class 10; U3; V30; A1):

https://www.microcenter.com/product/644261/Premium_64GB_microSDXC_Card_UHS-I_Flash_Memory_Card_C10_U3_V30_A1_Micro_SD_Card_with_Adapter_-_2_Pack

I'm beating up the system with reads again to see if I can trigger it.

[aperezbios]

@vom513 I personally wouldn't expect much from a $9 white-label/store brand 64GB consumer grade microSD card, for the record. There's no way to know who made it, to what quality standard it was made, etc. Most consumer SD cards mostly focus on doing sequential, high throughput reads and writes. If you want to try against a different low-cost 64GB SD card, I'd recommend trying this one

[vom513]

Point taken. Just ordered a few. Nevertheless - I still want to get a full debug to you guys in case something else is going on. Thanks.

[vom513]

Well the machine certainly crashed - but I wasn't watching it in realtime. Here is an actual debug log finally.

It's > 100mb so I put it up on my domain so you can grab it:

https://burn.net/zululog-ss10-watchdog.txt

[PetteriAimonen]

Thanks. Similar symptoms but this time hanging while transferring status byte to host machine. To me this seems like the host machine might be randomly crashing for some reason, which then causes the SCSI transfer to stop also, resulting in timeout on ZuluSCSI side. It can still be SCSI-related, but more likely to be on the host interface side (ISA/PCI or other system IO bus).

In general if it was a problem on the SCSI bus itself, the host would show IO error or just retry the request, instead of crashing.

[aperezbios]

Thanks. Because this file is plain text, it compresses down to eleven megabytes: zululog-huge.zip