iotlab-testbed: serial_aggregator breaks the shell

[cgundogan]

When conducting an experiment on the iotlab testbed and using make iotlab-term OR the socat method described in [1], the shell breaks for some nodes after specifying some amount of commands.

I can only reproduce this when using experiments with a fairly large amount of nodes (I tested with 40). I could not reproduce this for an experiment with 10 nodes.

Reproducable by doing the following:

1. connect via make iotlab-term or [1]
2. enter command ifconfig and fibroute in an alternating fashion
3. see that for some nodes the command fibroute may yield the output of ifconfig
4. after observing this behaviour for a specific node, this node's id can be used to talk to it directly by prefixing the command with node_id;<command>. One can see that any command will be issued to this node with a "delay" in a sense, that the previous command is executed first and the current command will most likely be executed as the next command.

I am not sure if this is a bug on RIOT's side (shell robustness) or a misbehaviour of the serial_aggregator. Therefore, I just use the label: bug.

[1] https://www.iot-lab.info/tutorials/nodes-serial-link-aggregation/#Give_access_as_a_tcp_socket

EDIT: all tests were conducted with ng_networking on master.

[cgundogan]

@cladmi this may also be of interest to you

[jnohlgard]

My first hunch would be to check if there is some buffering going on inside pyterm or whatever is used on iot-lab and/or the other tools that cause the end of the command line to stay in the buffer and not sent out on the serial port until more text arrives. It is less likely, however possible, that the RIOT shell would behave like this on random nodes with the same binary flashed.

[cgundogan]

Further details: Once I enforce such a misbehaviour on a node, I bypass the serial aggregator by connecting via ssh to the testbed and use nc <node_id> 20000. I get the same results as with the serial aggregator, commands are issued with a delay.

@gebart buffering might be a/the problem, but unfortunately I am not educated enough in the internals of the testbed.
@cladmi may be able to enlighten us?

[cgundogan]

@gebart I tried to execute the reboot statement by issueing it several times until it gets recognized by the shell. After a reboot, this node handles shell commands correctly again. So the problem lies definitely in the robustness of the RIOT shell. At least to a certain degree, because this only happens in the interplay when using the serial aggregator and a bunch of nodes in the experiment (>~30).

[OlegHahm]

Have you checked if the problem is not simply related to still using uart0?

[OlegHahm]

Try with #3402 or #3555.

[cgundogan]

I tried with #3402. I am not sure if it was due to the shell modification, but some nodes started to ignore my shell input after playing around some time on the iotlab with the serial aggregator. When I looked at such a particular node without the serial aggregator, then I was endlessly receiving the prompt character >.

[OlegHahm]

Can you try with pyterm?

[cgundogan]

Can you try with pyterm?

Could you elaborate on this?

[OlegHahm]

You wrote that you encountered these problems with serial_aggregator and I wanted to check if this reproducible with pyterm as well.

[cladmi]

One issue that can come from serial_aggregator is that it's only handling lines. Until a line break arrives, the content is kept in a buffer. https://github.com/iot-lab/aggregation-tools/blob/master/iotlabaggregator/serial.py#L106

Another issue I just discovered, is that maybe I should run the 'utf-8' decode on data before splitting the lines. This can, maybe, lead to issues with 'splitlines' not splitting if it gets some random characters, like in case of a crash or a reboot for example. But in that case decoding issues will need to be handled correctly.

If this is the reason for the problems, I will look into it.

[OlegHahm]

@cgundogan; I updated https://github.com/OlegHahm/RIOT/tree/shell_uart0_newlib_distinction - should fix the issue for now.

[OlegHahm]

@cgundogan, have you encountered the problem recently?

[cgundogan]

Tested on the iotlab with 50 nodes. Works 👍