Tasks/1019 flowetl debugging test failures

[danwilliams]

Closes #1019

I have:

• Formatted any Python files with black
• Brought the branch up to date with master
• Added any relevant Github labels
• Added tests for any new additions
• Added or updated any relevant documentation
• Added an Architectural Decision Record (ADR), if appropriate
• Added an MPLv2 License Header if appropriate
• Updated the Changelog

Description

This introduces the FLOWETL_INTEGRATION_TESTS_SAVE_AIRFLOW_LOGS variable in an attempt to make debugging failures easier. It will save the logs to /mounts/logs mounted on the host and also outputs all logs using Python's logging (although, this logging output is not really well formatted - not sure how it could be made better, it currently just cats all the logs, outputting the result).

Having considered the other options as well, it seems like most (if not all) failures would be evident from the airflow logs; navigating through the Airflow's UI might be nicer but getting all the logs should be the crucial thing.

[codecov]

Codecov Report

Merging #1066 into master will increase coverage by 1.09%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #1066      +/-   ##
==========================================
+ Coverage   94.12%   95.21%   +1.09%     
==========================================
  Files         186      184       -2     
  Lines        8300     8188     -112     
  Branches      784      778       -6     
==========================================
- Hits         7812     7796      -16     
+ Misses        347      281      -66     
+ Partials      141      111      -30

Flag	Coverage Δ
#autoflow_unit_tests	92.84% <ø> (?)
#flowapi_unit_tests	83.97% <ø> (ø)	⬆️
#flowauth_unit_tests	93.68% <ø> (ø)	⬆️
#flowclient_unit_tests	78.78% <ø> (ø)	⬆️
#flowetl_unit_tests	97.35% <ø> (ø)	⬆️
#flowkit_jwt_generator_unit_tests	?
#flowmachine_unit_tests	90.95% <ø> (ø)	⬆️
#integration_tests	69.69% <ø> (+0.04%)	⬆️

Impacted Files	Coverage Δ
flowkit_jwt_generator/flowkit_jwt_generator/jwt.py
...it_jwt_generator/flowkit_jwt_generator/fixtures.py
flowmachine/flowmachine/core/cache.py	97.18% <0%> (+1.4%)	⬆️
autoflow/autoflow/model.py	100% <0%> (+9.09%)	⬆️
autoflow/autoflow/workflows.py	100% <0%> (+9.09%)	⬆️
autoflow/autoflow/utils.py	100% <0%> (+10.44%)	⬆️
autoflow/autoflow/sensor.py	100% <0%> (+16.66%)	⬆️
autoflow/autoflow/parser/parser.py	100% <0%> (+18.18%)	⬆️
autoflow/autoflow/parser/workflow_schema.py	96.42% <0%> (+21.42%)	⬆️
autoflow/autoflow/parser/custom_fields.py	100% <0%> (+25%)	⬆️
... and 5 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 203f258...8e345ab. Read the comment docs.

[cypress]

---

Test summary

57 • 0 • 0 • 0

---

Run details

Project	FlowAuth
Status	Passed
Commit	8e345ab
Started	Dec 12, 2019 10:45 AM
Ended	Dec 12, 2019 10:48 AM
Duration	02:29 💡
OS	Linux Debian - 8.11
Browser	Electron 73

View run in Cypress Dashboard ➡️

---

This comment has been generated by cypress-bot as a result of this project's GitHub integration settings. You can manage this integration in this project's settings in the Cypress Dashboard

[jc-harrison]

LGTM

Might be good to use this in CI to make the logs available, but not essential right now.