Use YAML for metadata and testcases

[nashif]

Make sanitycheck support YAML instead of ini and fetch data from boards and
samples to help with better test coverage.

---

This change is 

[pfalcon]

Well, YAML is more expressive than .ini, assuming there is a reason to use that expressiveness already, looks good ;-).

[pfalcon]

looks good ;-).

But likely postpone until 1.9 window opens anyway, I guess?

[inakypg]

if identifier and name are always the same, loose one.
If it is always goign to look like the file name, default to the file name instead of having yet another field.

[nashif]

its not the same, I did most of the boards, not all of them though. I do not want to rely on file names, we have a structured syntax, why not use that, otherwise it will be confusing and we will be maintaining information in different ways.

[inakypg]

same thing though then, you'll have the same name on the board directory or the file name and two identifiers. Things will get confusing.

boards/arc/arduino_101_sss/arduino_101_sss.yaml
@@ -0,0 +1,12 @@
+identifier: arduino_101_sss
+name: arduino_101_sss

Four repetitions of the same name. I would default it like:

with open(this_filename) as f:
  y = yaml.load(f.read())
if os.path.basename(this_filename) == "board.yaml":
  base_id = os.path.basename(os.path.dirname(this_filename))  # get the directory name
identifier = y.get('identifier', base_id)
name = y.get('name', identifier)

this way you can avoid repetition while at the same time allowing an override

[nashif]

we have cases where might have multiple board definitions inside a board directory, so that already does not work.

[inakypg]

loose name in favour of the file name sans .yaml or the directory (blinky)

[inakypg]

tests/test_arm

remove duplicated "test"

[inakypg]

suggest loosing this build_only and make it a tag (eg: tags: build_only, slow, etc) instead of adding fields

[pfalcon]

I'd say build_only is important enough to stand on its own.

[inakypg]

how do we figure out at build time which platform has which RAM?

[inakypg]

tags crypto aes ccm
test path crypto/ccm

We need to have the system pickup tags from the path, as there is info there that we don't need to repeat (it is an initial classification after all)

[pfalcon]

That will lead to magic behavior nobody understands. "Explicit is better than implicit".

[nashif]

agree with @pfalcon , this is also not part of what this change is trying to do.

[inakypg]

I tend to agree in general with @pfalcon , but there is so much duplication in all of this with the amount of TCs that at some point this going to be out of sync and will be also confusing. There is the possibility of having a script that tries to ensure the "path" and the "tags" kinda match or calls upon the ones that don't.

[nashif]

Moving things around and copying things to a different path makes this very difficult to manage and all of the sudden a test case that appears in a report as XYZ is not ABC because the path changed. I do not see this as duplication, the "database" is the file, not its location in the git tree.

[inakypg]

Got some spec on what each different yaml is? I'd like to see different names so it is easy to tell if it is a board descriptor vs a TC descriptor.