Python test improvements

[amdecker]

Addresses

• Fix test against cache for python #376

Changes

• Python test does not fail on expected failures
• Python test console log level is now "INFO" by default, but it can be changed from the console [EDIT by GLT]

python tests/test_against_cache.py -l warning

• Assertion now gives shot and column of failure e.g. AssertionError: Comparison failed on shot 1150805012, column sxr

[gtrevisan]

your last commits introduces this newline, but it breaks stuff for me with:

Traceback (most recent call last):
  File "disruption-py/tests/test_against_cache.py", line 18, in <module>
    from tests.utils.eval_against_sql import (
  File "disruption-py/tests/utils/eval_against_sql.py", line 188
    ), f"Comparison failed on shot {data_difference.shot_id}, column {
       ^
SyntaxError: unterminated string literal (detected at line 188)

did the newline solve stuff for you? which python version are you using?

[gtrevisan]

for C-MOD, I have 35 passed, 11 xfailed for pytest, but a lot of Expected failure and succeeded for python -- is that expected?

on a separate note, now that you added a quick shortcut to switch the log-level from the cli, I'll put INFO back as default.

[amdecker]

for C-MOD, I have 35 passed, 11 xfailed for pytest, but a lot of Expected failure and succeeded for python -- is that expected?

I took a look and this happens because we expect failure by column rather than shot and column. For a given column, some shots pass the test and some shots fail. If we succeeded for all shots, pytest would mark as xsuccess, but as soon as one shot fails it marks xfail.