Flowetl integration test failures are difficult to debug

[maxalbert]

Example: https://circleci.com/gh/Flowminder/FlowKit/73699

This test run fails because there is an interpolation error in the sql templates which results in events.{cdr_type} being interpolated as events.CDRType.CALLS instead of events.calls.

However, this is only apparent when clicking through the Airflow UI - which in turn is only possible when re-running the failed CircleCI job with ssh enabled while also forwarding port 28080:

ssh -L 28080:localhost:28080

With this the Airflow UI can be inspected on http://localhost:28080 while the job is running on CircleCI, but of course this is a less than ideal debugging experience.

An alternative is to try and run the integration tests locally. However, currently the flowetl integration tests are set up to pull the docker images before spinning them up for the integration test runs, and this ignores any local changes to the code. In order to be able to run the integration tests on the local version of the code the pull_docker_images fixture needs to be disabled and the flowetl docker image needs to be built locally (tagged as latest, to match the value FLOWETL_TESTS_CONTAINER_TAG=latest) before running the tests.

All of this illustrates that the flowetl functionality is fairly heavily coupled to the docker image & running docker container, which is less then ideal but for the most part unavoidable due to the use of Airflow.

[greenape]

Ugh. Can we not just run against airflow outside the container? Or would that not help? Would this have been evident from airflow's logs?

[danwilliams]

Ugh. Can we not just run against airflow outside the container? Or would that not help? Would this have been evident from airflow's logs?

It looks like most (if not all) failures would be evident from the logs. It would be nice to navigate through the Airflow's UI, but on CI this might not be the best idea, the solution implemented here is to get all the logs generated by airflow before killing the container.