Tests listeted in test-manifest contain C specific names

[cmnrd]

Some of the tests listed in test-manifest end with a 'C' which point to a C specific test. I think the 'C' should be removed in all tests to have the same tests across targets. Or is there some other meaning to the 'C' that I don't see?

It might also be beneficial in the future to add the possibility of defining additional target specific tests.

[edwardalee]

I have renamed the C tests to remove the C. Also changed them so that main reactors named Main have been renamed to match the filename. Should I also change the C++ tests?

[lhstrh]

All targets now use the same manifest, so the names in the manifest must match those in both the c and cpp directory. We’ll have to figure out a way to add target-specific tests. My suggestion would be to partition the manifest file with headers. Marten

On Thu, Nov 14, 2019 at 11:38 AM Edward A. Lee ***@***.***> wrote: I have renamed the C tests to remove the C. Also changed them so that main reactors named Main have been renamed to match the filename. Should I also change the C++ tests? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#55>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEYD47AW7CG5RKVRFU76X3DQTWSKPANCNFSM4JNNYVXA> .

--

…

-- Marten Lohstroh, MSc. | Ph.D. Candidate University of California | 545Q Cory Hall Berkeley, CA 94720 | +1 510 282 9135

[lhstrh]

To allow target specific tests, here are three proposals:

• rename test-manifest to common-test-manifest and introduce <target>-test-manifest files to list target-specific tests
• keep test-manifest and introduce <target>/test-manifest files to list target-specific tests
• keep test-manifest add sectioning to the file to denote target-specific tests
  Not sure which one to choose. Comments?

[cmnrd]

I am in favour of keeping one single file with multiple 'sections'. But I would suggest to make the test-manifest a configuration file with a slighlty more complex structure. This would also help us to solve the problem discussed in 47aff87 and #54. The configuration file would be the central place to describe all tests and if we change smth in the way tests are handled, we would only need to modify the test script and adapt the configuration where needed. We could als include support for expected fails (i.e. to validate compile time or runtime checks) or expected results.

common:
  HelloWorld:
    timeout: 1
    fast: false
C:
  OnlyCTest:
    timeout: 2
    fast: true
C++:
  CppExpectedFail:
    timeout: 1
    fast: false
    expected_fail: "true"

This of course would require run-lf-tests to be a more complex script, but it shouldn't be hard to implement something like that in python or another scripting language. If you want I could give this a shot, but I am not sure yet when I will find time for it.

[schoeberl]

Before inventing your own syntax for the configuration, I would propose YAML [1]. YAML parsers should be available in many languages. Martin [1] https://en.wikipedia.org/wiki/YAML

…

On 26 Nov, 2019, at 9:54, Christian Menard ***@***.***> wrote: I am in favour of keeping one single file with multiple 'sections'. But I would suggest to make the test-manifest a configuration file with a slighlty more complex structure. This would also help us to solve the problem discussed in 47aff87 <47aff87> and #54 <#54>. The configuration file would be the central place to describe all tests and if we change smth in the way tests are handled, we would only need to modify the test script and adapt the configuration where needed. We could als include support for expected fails (i.e. to validate compile time or runtime checks) or expected results. common: HelloWorld: timeout: 1 fast: false C: OnlyCTest: timeout: 2 fast: true C++: CppExpectedFail: timeout: 1 fast: false expected_fail: "true" This of course would require run-lf-tests to be a more complex script, but it shouldn't be hard to implement something like that in python or another scripting language. If you want I could give this a shot, but I am not sure yet when I will find time for it. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#55>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAE63GETPVW2LEU4A6RNPU3QVTP3TANCNFSM4JNNYVXA>.

[cmnrd]

Agreed. In fact the code snippet I showed is YAML. I only forgot to mention that.

[cmnrd]

I think this is closed by the organization of our test directories and the new JUnit based test infrastructure.