[AIRFLOW-4116] Multi-staging includes CI image [Step 2/3]

[potiuk]

Make sure you have checked all steps below.

Jira

• My PR addresses the following Airflow Jira

Description

• Here are some details about my PR, including screenshots of any UI changes:

This is 2nd stage of parent https://issues.apache.org/jira/browse/AIRFLOW-3718 task. It implements CI image and add custom hook/build script that builds both images in parallel.

Tests

• Tests will be added in the third stage of the change where Travis CI will be switched to use those images for testing.

Commits

• My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters (not including Jira issue reference)
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

Documentation

No documentation needed for that change. Design is documented in https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-10+Multi-layered+and+multi-stage+official+Airflow+image

Follow up documentation will be added in Simplified Development Workflow change: #4932

Code Quality

• Passes flake8

[codecov-io]

Codecov Report

Merging #4937 into master will increase coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #4937      +/-   ##
==========================================
+ Coverage   79.02%   79.02%   +<.01%     
==========================================
  Files         481      481              
  Lines       30203    30203              
==========================================
+ Hits        23868    23869       +1     
+ Misses       6335     6334       -1

Impacted Files	Coverage Δ
airflow/models/taskinstance.py	92.61% <0%> (+0.17%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 02ef974...c25a27e. Read the comment docs.

[gerardo]

This file is also in #4936

[potiuk]

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Note to reviewer: Please review only the last commit (use commits section above). This change is based on yet unmerged #4936 change and it contains also commit from that change. It's a github limitation unfortunately.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

[potiuk]

Hey @ashb @Fokko - I think that commit should be pretty good for further steps and raising the AIP on the devlist? What do you think? Do you have any further comments?

[Fokko]

I understand. I’ll look into it tomorrow!

[Fokko]

Looking great @potiuk

I have some comments about esthetical and about version control, would be great to resolve these. Apart from that: LGTM.

[potiuk]

Thanks for the feedback @Fokko and @ashb - I addressed I think all of the feedback. There is one remaining query @ashb - about separating airflow sources from the dag/logs files - see the unresolved conversation above.. I will run the last round of tests/CI builds with it and will merge it as soon as I got it all tested if you are ok with my proposal.

[ashb]

@potiuk Looking good now.

#4937 (comment) I think is resolved?

Last two outstanding issues are

• [AIRFLOW-4116] Multi-staging includes CI image [Step 2/3] #4937 (comment); and
• [AIRFLOW-4116] Multi-staging includes CI image [Step 2/3] #4937 (comment)

Right? Step 3/3 will be to actually change our CI to use these images right?

[potiuk]

@ashb -> All resolved now except changing potiuk->airflow. I am flying back home from the Airflow Bay Area meetup (which was awesome BTW) and once I get pass the jetlag I will merge it and proceed with fixing last test failure in 3/3 and we can hopefully close the AIP and proceed to Simplified Development Workflow. I had fantastic response from the people at the meetup about Breeze :)

[Fokko]

Nice, great work @potiuk