tests: init RobotFramework and add basic xtimer test #10843

smlng · 2019-01-22T12:55:10Z

Contribution description

This introduces common files to enable RobotFramework based (HIL) tests in RIOT.
It also ships a very basic xtimer test application utilising the RobotFramework.
The latter will be extended later on, but should give others a starting point to see how to
use RF with RIOT.

RF tests (currently) run from a distinct make target, i.e. robot-test; so will not interfere with existing tests. This might be changed later on, but for now it should ease introduction of RF tests w/o breaking existing stuff.

Testing procedure

Currently this only works for real boards connected via serial, so connect you favourite board and run

BOARD=<name> make -C tests/xtimer_cli flash robot-test

Issues/PRs references

replaces testing: introduce RobotFramework tests #10095
partly implements stuff from doc/rdm: RDM on a common api for testing #10624

Provides common files, directory structure and make targets to enable (HIL) tests based on the RobotFramework.

MichelRottleuthner

Looks good to me. Only some non-blocking comments below.
edit: I also did a quick test on esp8266 and got the beautiful robot framework output

MichelRottleuthner · 2019-01-22T13:28:50Z

tests/xtimer_cli/README.md

+## Requirements
+
+The automated tests are based on RobotFramework and thus depend on additional
+Python packages, namely `robotframework` and `riot_pal`.


maybe add a reference to dist/robotframework/README.md for installing the requirements here?

yep makes sense

MichelRottleuthner · 2019-01-22T13:29:28Z

tests/xtimer_cli/main.c

@@ -0,0 +1,62 @@
+/*
+ * Copyright (C) 2018 HAW Hamburg


it's an old one

MichelRottleuthner · 2019-01-22T13:33:20Z

tests/xtimer_cli/main.c

+
+    return 0;
+}
+


Didn't look it up, but is the echo cmd described in #10624 already used by riot_pal? If yes, maybe add it here?

nope, no echo there, and I would rather make (as discussed offline) a PR introducing both (metadata and echo) as a shell module, and not start adding stuff here that is not needed. I need the metadata but I don't need or use echo right now, so why bother.

MichelRottleuthner · 2019-01-22T13:41:23Z

tests/xtimer_cli/tests/01__xtimer_base.robot

+Suite Setup         Run Keywords    Reset Application
+...                                 DUT Must Have API Firmware
+Test Setup          Run Keywords    Reset Application
+...                                 DUT Must Have API Firmware


As this is somehow a POC/showcase for using robot framework a little more comments here would be nice to make clear what Suite Setup, Test Setup and so on actually does.
Also IIRC the second DUT Must Have API Firmware is a little hack to ensure the communication channel is working correctly. -> this should be noted explicitly to make it clear to others that this is required.

first: why should I repeat the RF documentation? There is plenty documentation on Suite/Test Setup at the original sources, I can add some links in the dist/robotframework/README.md but copy-pasting docu doesn't make sense (to me).

second: no that's no hack in that sense, it's just there to ensure that before each test the devices are reset and responding correctly before running the tests. The Suite Setup could be avoided but on the other it helps to skip all tests if there is a problem with the board.

aabadie

Blocking to avoid unexpected merge.

tcschmidt · 2019-01-22T15:24:07Z

Blocking to avoid unexpected merge.

What is it you want to question or change?

aabadie · 2019-01-22T15:30:14Z

What is it you want to question or change?

Since I'm not aware of any conclusion with #10241 (where the discussion is not going toward the direction of using the Robot Framework IMHO), I don't want to have this merged by mistake before the next release.

tcschmidt · 2019-01-22T15:41:08Z

What is it you want to question or change?

Since I'm not aware of any conclusion with #10241 (where the discussion is not going toward the direction of using the Robot Framework IMHO), I don't want to have this merged by mistake before the next release.

The conclusion from #10241 to me seems that there is rough consensus on going ahead with Robot at least for the HIL testing - along with some experienced guidance by @pekkanikander. In case there are further arguments or comments regarding the framework itself, they should be added to #10241.

smlng · 2019-01-23T09:51:23Z

addressed comments by @MichelRottleuthner

cladmi · 2019-01-23T13:23:57Z

This fails for me:

pip install --upgrade --user -r dist/robotframework/requirements.txt
Collecting robotframework (from -r dist/robotframework/requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/36/c6/6f89c80ac5a526a091bd383ffdfc64c9a68d9df0c775d4b36f03d8e0ac25/robotframework-3.1.1-py2.py3-none-any.whl
Collecting riot_pal==0.2.2 (from -r dist/robotframework/requirements.txt (line 2))
  Could not find a version that satisfies the requirement riot_pal==0.2.2 (from -r dist/robotframework/requirements.txt (line 2)) (from versions: )
No matching distribution found for riot_pal==0.2.2 (from -r dist/robotframework/requirements.txt (line 2))

cladmi · 2019-01-23T13:26:07Z

Hmm, my pip by default used python2 but it works with pip3.

MrKevinWeiss · 2019-01-23T13:28:32Z

yup, not supporting python2 😞 I think it specifies it in the pip package.

cladmi · 2019-01-23T13:49:19Z

As our target is ubuntu stable, pip and python are by default python2 so the doc should match this.

Currently this only works for real boards connected via serial, so connect you favourite board and run

So it does not work with native but also does not work with any board using stdio_rtt.
So no ruuvitag, hamilton (pr #9013 ), thingy52.


python3 -m robot.run \
                        --name tests_xtimer_cli \
                        --noncritical warn-if-failed \
                        --settag "APP_tests_xtimer_cli" \
                        --settag "BOARD_hamilton" \
                        --metadata RIOT-Version:2018.10-RC1-1121-g2dc8e5-pr/riot/10843/init_RobotFramework_and_add_basic_xtimer_test \
                        --metadata RIOT-Board:hamilton \
                        --metadata RIOT-Application:tests_xtimer_cli \
                        --xunit xunit.xml \
                        -P "/home/harter/work/git/RIOT/tests/xtimer_cli/tests:/home/harter/work/git/RIOT/dist/robotframework/lib:/home/harter/work/git/RIOT/dist/robotframework/res:/home/harter/work/git/RIOT/dist/tests" \
                        -d /home/harter/work/git/RIOT/build/robot/hamilton/tests_xtimer_cli/ \
                        tests/01__xtimer_base.robot
[ ERROR ] Error in file '/home/harter/work/git/RIOT/tests/xtimer_cli/tests/01__xtimer_base.robot': Environment variable '%{BAUD}' not found. Did you mean:
    %{BOARD}
==============================================================================
tests_xtimer_cli :: Basic tests to verify functionality of the Xtimer API.
==============================================================================
Xtimer Now Should Succeed :: Verify xtimer_now() API call.            | FAIL |
Parent suite setup failed:
No keyword with name 'Get Metadata' found.
------------------------------------------------------------------------------
Xtimer Values Should Increase :: Compare two xtimer values (t1, t2... | FAIL |
Parent suite setup failed:
No keyword with name 'Get Metadata' found.
------------------------------------------------------------------------------
tests_xtimer_cli :: Basic tests to verify functionality of the Xti... | FAIL |
Suite setup failed:
No keyword with name 'Get Metadata' found.

2 critical tests, 0 passed, 2 failed
2 tests total, 0 passed, 2 failed
==============================================================================
Output:  /home/harter/work/git/RIOT/build/robot/hamilton/tests_xtimer_cli/output.xml
XUnit:   /home/harter/work/git/RIOT/build/robot/hamilton/tests_xtimer_cli/xunit.xml
Log:     /home/harter/work/git/RIOT/build/robot/hamilton/tests_xtimer_cli/log.html
Report:  /home/harter/work/git/RIOT/build/robot/hamilton/tests_xtimer_cli/report.html
/home/harter/work/git/RIOT/tests/xtimer_cli/../../makefiles/robotframework.inc.mk:8: recipe for target 'robot-test' failed
make: *** [robot-test] Error 2
make: Leaving directory '/home/harter/work/git/RIOT/tests/xtimer_cli'

Please do no re-invent the wheel and use make term as a base.

tcschmidt · 2019-01-27T13:36:31Z

Since I'm not aware of any conclusion with #10241 (where the discussion is not going toward the direction of using the Robot Framework IMHO), I don't want to have this merged by mistake before the next release.

It's feature-freeze now, so no danger to sneak into the release: You may want to remove the operational block then @aabadie ?

aabadie · 2019-01-27T14:03:42Z

so no danger to sneak into the release: You may want to remove the operational block

Sorry, but no. My main blocker point is still not full-filled: there's no consensus whether we add and maintain RF files (and all the custom keywords that will come with) in the RIOT repository or not.

tcschmidt · 2019-01-27T15:30:22Z

Sorry, but no. My main blocker point is still not full-filled: there's no consensus whether we add and maintain RF files (and all the custom keywords that will come with) in the RIOT repository or not.

There is no open issue nor argument, if I'm not mistaken. IMO this is considered rough consensus, even if some people like the solution more and some less. Rough consensus does not require everybody to be enthusiastic, otherwise it would be called euphoria.

Regarding the repository: Are you suggesting that RF testing should be a separate Repo within the RIOT organization? Currently, we are following the common RIOT logic (as done with Murdoc etc.): Framework specific scripts and frontend stuff like stylesheets are separate repos, the actual tests are located in the RIOT repo.

cladmi · 2019-01-28T13:30:08Z

My main remarks from #10241 (comment) where not addressed.

It is going in a direction to put the abstraction code as a RobotFramework specific class https://github.com/RIOT-OS/RIOT/pull/10843/files#diff-2756e48a9944cf93cbea221c7c933574
So forces into this scheme anyway.

And it also does not reuse the normal RIOT way to access the serial port which is make term and so fails with native, IoT-LAB integration (or any node access through ssh like the Hamburg HIL nodes) and boards that cannot be accessed directly through only a serial. So it does not interact correctly with RIOT and is really specific to your own setup.

Until both are addressed, I still consider it as harmful to have as it will push toward a split in the tests implementation where any improvement one one side do not benefit the other.

smlng · 2019-01-28T14:49:45Z

It is going in a direction to put the abstraction code as a RobotFramework specific class https://github.com/RIOT-OS/RIOT/pull/10843/files#diff-2756e48a9944cf93cbea221c7c933574
So forces into this scheme anyway.

That small python wrapper for Xtimer is RF related indeed, at least the following lines are required to handle the imports correctly and auto-generate keywords in RF:

    ROBOT_LIBRARY_SCOPE = 'TEST SUITE'
    ROBOT_LIBRARY_VERSION = get_version()

However, this does not mean this class/lib cannot be used by another framework. Is there a (simple) way to make the imports and those two variables conditional, i.e. only have them when this file is used with RF?

And it also does not reuse the normal RIOT way to access the serial port which is make term and so fails with native, IoT-LAB integration (or any node access through ssh like the Hamburg HIL nodes) and boards that cannot be accessed directly through only a serial. So it does not interact correctly with RIOT and is really specific to your own setup.

I agree, and I'm working on changing this to work with make term and thus also allow to use it with IoT-Lab.

tcschmidt · 2019-01-28T17:15:30Z

See related PR: RIOT-OS/riot_pal#24

MrKevinWeiss

To keep it simple see the following comments

MrKevinWeiss · 2019-01-29T16:20:31Z

tests/xtimer_cli/tests/Xtimer.py

@@ -0,0 +1,63 @@
+# Copyright (C) 2018 Kevin Weiss <[email protected]>


MrKevinWeiss · 2019-01-29T16:20:38Z

tests/xtimer_cli/tests/Xtimer.py

+import logging
+
+from riot_pal import DutShell
+from robot.version import get_version


Also this should be separated out into a robot independent interface then a simple wrapper for robot.

MrKevinWeiss · 2019-01-29T16:21:40Z

dist/robotframework/lib/PhilipAPI.py

+from robot.version import get_version
+from time import sleep
+
+class PhilipAPI(LLMemMapIf):


Maybe we start without philip at all and just do non-philip tests to show as an example.

MrKevinWeiss · 2019-01-29T16:22:09Z

dist/robotframework/res/philip.keywords.txt

@@ -0,0 +1,11 @@
+*** Settings ***
+Library             PhilipAPI  port=%{PHILIP_PORT}  baudrate=${115200}  WITH NAME  PHILIP


maybe remove for now...

smlng · 2019-02-21T11:08:51Z

closing as this moved to separate repo for faster integration and progression, see https://github.com/RIOT-OS/RobotFW-tests

smlng added 2 commits January 22, 2019 13:35

tests: initial robotframework integration

d8b9a9a

Provides common files, directory structure and make targets to enable (HIL) tests based on the RobotFramework.

tests: add initial xtimer robot test

f7282bc

smlng added Area: tests Area: tests and testing framework Type: new feature The issue requests / The PR implemements a new feature for RIOT CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR labels Jan 22, 2019

smlng requested review from MrKevinWeiss and MichelRottleuthner January 22, 2019 12:55

This was referenced Jan 22, 2019

tests: I2C HIL tests with RobotFramework #10774

Closed

testing: introduce RobotFramework tests #10095

Closed

MichelRottleuthner approved these changes Jan 22, 2019

View reviewed changes

aabadie requested changes Jan 22, 2019

View reviewed changes

fixup! tests: add initial xtimer robot test

3a5439f

MrKevinWeiss reviewed Jan 29, 2019

View reviewed changes

smlng closed this Feb 21, 2019

smlng deleted the pr/robot/init_xtimer branch February 21, 2019 11:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tests: init RobotFramework and add basic xtimer test #10843

tests: init RobotFramework and add basic xtimer test #10843

smlng commented Jan 22, 2019

MichelRottleuthner left a comment •

edited

Loading

MichelRottleuthner Jan 22, 2019

smlng Jan 22, 2019

MichelRottleuthner Jan 22, 2019

smlng Jan 22, 2019

MichelRottleuthner Jan 22, 2019

smlng Jan 22, 2019

MichelRottleuthner Jan 22, 2019

smlng Jan 22, 2019

aabadie left a comment

tcschmidt commented Jan 22, 2019

aabadie commented Jan 22, 2019

tcschmidt commented Jan 22, 2019 •

edited

Loading

smlng commented Jan 23, 2019

cladmi commented Jan 23, 2019

cladmi commented Jan 23, 2019

MrKevinWeiss commented Jan 23, 2019

cladmi commented Jan 23, 2019 •

edited

Loading

tcschmidt commented Jan 27, 2019

aabadie commented Jan 27, 2019

tcschmidt commented Jan 27, 2019

cladmi commented Jan 28, 2019

smlng commented Jan 28, 2019

tcschmidt commented Jan 28, 2019

MrKevinWeiss left a comment

MrKevinWeiss Jan 29, 2019

MrKevinWeiss Jan 29, 2019

MrKevinWeiss Jan 29, 2019

MrKevinWeiss Jan 29, 2019

smlng commented Feb 21, 2019

		@@ -0,0 +1,63 @@
		# Copyright (C) 2018 Kevin Weiss <[email protected]>

		@@ -0,0 +1,11 @@
		* Settings *
		Library PhilipAPI port=%{PHILIP_PORT} baudrate=${115200} WITH NAME PHILIP

tests: init RobotFramework and add basic xtimer test #10843

tests: init RobotFramework and add basic xtimer test #10843

Conversation

smlng commented Jan 22, 2019

Contribution description

Testing procedure

Issues/PRs references

MichelRottleuthner left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aabadie left a comment

Choose a reason for hiding this comment

tcschmidt commented Jan 22, 2019

aabadie commented Jan 22, 2019

tcschmidt commented Jan 22, 2019 • edited Loading

smlng commented Jan 23, 2019

cladmi commented Jan 23, 2019

cladmi commented Jan 23, 2019

MrKevinWeiss commented Jan 23, 2019

cladmi commented Jan 23, 2019 • edited Loading

tcschmidt commented Jan 27, 2019

aabadie commented Jan 27, 2019

tcschmidt commented Jan 27, 2019

cladmi commented Jan 28, 2019

smlng commented Jan 28, 2019

tcschmidt commented Jan 28, 2019

MrKevinWeiss left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

smlng commented Feb 21, 2019

MichelRottleuthner left a comment •

edited

Loading

tcschmidt commented Jan 22, 2019 •

edited

Loading

cladmi commented Jan 23, 2019 •

edited

Loading