Sync v2-5-stable with v2-5-test to release 2.5.1 #28883
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
(cherry picked from commit 24745c7)
Previously if get_uri was called it would fail with `NoneType not iterable`, because of the check `if '-' in conn_type`. (cherry picked from commit b124d6c)
Co-authored-by: doiken <[email protected]> (cherry picked from commit e350b6d)
(cherry picked from commit 76f81cd)
* dagrun, next_dagruns_to_examine, add MySQL index hint * resolve static check errors (cherry picked from commit 5e45cb0)
* Make sure we can get out of a faulty scheduler state This PR fixed the case where we have a faulty state in the database. The state that is fixed is that both the unmapped task instance and mapped task instances exist at the same time. So we have instances with map_index [-1, 0, 1]. The -1 task instances should be removed in this case. (cherry picked from commit 73d9352)
(cherry picked from commit d97a249)
(cherry picked from commit 820c5bb)
The fix here was to set changed_tis to True if there was an expansion. (cherry picked from commit f89ca94)
Co-authored-by: Igor Kholopov <[email protected]> (cherry picked from commit 4a39115)
(cherry picked from commit 405087d)
* fix merging connected dataset graphs * refactor graph calculation (cherry picked from commit f1c4c27)
I noticed a case where the same local providers were loaded more than once, and it turned out to be caused by having `.` in the python search path. The fix for this is to canonicalize the path before looking for providers in it, and not searching in a path more than once. (cherry picked from commit 1a02ad9)
(cherry picked from commit 56b5f3f)
(cherry picked from commit 893253a)
(cherry picked from commit 8924cf1)
(cherry picked from commit abe3b31)
(cherry picked from commit ada91b6)
Ensure that when a user clicks on 'Queue up new tasks' a list of tasks that will be run is returned. (cherry picked from commit af29ff0)
Calling `items` on config has the effect of calling `get` on each item. If we call `get` on a moved item, we will falsely get a warning letting us know to update our code. So, we suppress such warnings when iterating the config. (cherry picked from commit 2de6131)
Bumps [decode-uri-component](https://github.com/SamVerschueren/decode-uri-component) from 0.2.0 to 0.2.2. - [Release notes](https://github.com/SamVerschueren/decode-uri-component/releases) - [Commits](SamVerschueren/decode-uri-component@v0.2.0...v0.2.2) --- updated-dependencies: - dependency-name: decode-uri-component dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit c4cb5b3)
Warnings printed in CI have been making it difficult to see what is going on (they were taking far too much space after the test results and GitHub CI UI rendered those multi-line warnings slowly. Also we did not have the right tools to capture the number and list of warnings that we should deal with. We are usign pytest-capture-warnings plugin now that improves the situation twofold: * warning summary printed by the plugin in the output is shorter - each warning is at most one line * the warning text files are uploaded as artifacts which make them usable in any kind of approach where we want to attempt to start an effort to remove all warnings (cherry picked from commit 16fddba)
Tried fixing this before using simplefilter but it doesn't work when application threaded. See here https://docs.python.org/3/library/warnings.html#temporarily-suppressing-warnings. It was tricky to solve. When "actually" reading the values we call super().get. You'd think this would not invoke airflow config parser `get` right? But because of config parser interpolation, ultimately, it invokes `get` again on airflow config parser. So, we can manipulate an attr to signal when we are _accessing_ a deprecated key but only because we're retrieving the backcompat val (not because the user requested it). Additionally we have to handle the case where `items` is called (e.g. from within as_dict) which calls `get` for every option you have in your config. (cherry picked from commit 27a8463)
Co-authored-by: Tzu-ping Chung <[email protected]> (cherry picked from commit 56c0871)
* improve run/task changes from grid * fix confirm return type (cherry picked from commit a0c8502)
(cherry picked from commit 2c7bd92)
This PR performs housekeeping of the plugin examples: * makes the examples independent of Hive being installed * adds "has_access" in the examples * removes the misleading "metastore" (which is hive metastore not Airflow Metastore as used in other places This way our example will be much easier to apply by anyone. (cherry picked from commit 66eb282)
* Remove extra H1 & improve formatting of Listeners docs page I noticed that the documentation has an unclickable "Usage" page in the TOC. A little digging later, I discovered that this page contains an extra H1, and since this page is in the top level of the TOC, all the H1s on this page show up in the left docs sidebar. Demoted the "Usage" section to an H2, and fixed the other headers on this page to use consistent underlining with most other docs pages in this repo. I also took the liberty of sprucing up the language on the page to follow docs best practices, like shorter, highly readable sentences, title case in section titles, and bulleted lists to draw attention to important collections. * Remove extra newlines from specification discussion * Remove single newlines from listener API discussion * Remove nonexistent DagRun events from listeners page (cherry picked from commit 672264b)
ephraimbuddy
force-pushed
the
v2-5-test
branch
2 times, most recently
from
6b756c8 to
e54970c
Compare
…nd (#28261) The --integration tag should be standard flag rather than left for the common options. This change moves the flag to the right group. Images were regenerated because Rich does not know that the commands changed just when the option was moved to another group (this is a rich-click configuration and we are generating hash of commands from rich's command definition. As result of it, some of the breeze's svg files are changed. There are still subtle differences (mainly about font specification) on Linux and Maci and possibly it depends on what fonts are installed on your system - so when you regenerate images, font definition changes. This should be no issue in general as those images have the same hash and for all practical purposes, they are unchanged. (cherry picked from commit d8a0658)
Despite earlier attempts to fix the links there were still many places where the links were pointing to wrong versions of examples. It has been Noticed in: #27774 (comment) This change addresses it in three ways: * updated tool to fix the linx in historically released documentation (applied already in apache/airflow-site#706) * replaced hard-coded version with `|version|` everywhere and added sphinx extension to convert those to actual provider versions (though due to sphinx limitation it has to be done with post-processing of the generated .html rather than replacing during generation) * added pre-commit that detects any use of example_dags links to main/master/hard-coded version. (cherry picked from commit c8e348d)
…28673) The newly released Pygments 2.14.0 changed slightly the way it wraps tags around rendeered variables and tour test made far too many assumptions about the rendered output which started to fail on main with the new Pygments. This PR fixes it by actually testing only what it was supposed to test: - lack of the secret in rendered output - presence of *** masking in the output (cherry picked from commit 81cd6c7)
(cherry picked from commit 49e178e)
pierrejeambrun
force-pushed
the
v2-5-test
branch
from
e54970c to
6aa1360
Compare
The early cache is only needed when we push to main because it is only needed for PRs to the main brach. (cherry picked from commit 14783c6)
pierrejeambrun
force-pushed
the
v2-5-test
branch
from
6aa1360 to
449d6be
Compare
(cherry picked from commit 04306f1)
ephraimbuddy
force-pushed
the
v2-5-test
branch
from
449d6be to
ca3f70c
Compare
pierrejeambrun
approved these changes
Jan 13, 2023
eladkal
approved these changes
Jan 13, 2023
potiuk
approved these changes
Jan 13, 2023
(cherry picked from commit e0bb6be)
) (cherry picked from commit 518fd84)
This is the first stage of improving the way how integration tests are run in our CI - first we want to separate them in a separate packages, and then we want to run them separately - one integration each in the CI. (cherry picked from commit df608fe)
Previously we had separate integrations per service, but since we are moving the integration tests into separate job, it will be easier if the celery integration is a single one - this way we will have 1-1 relationship between tests to run and enabled integrations. The checks for various integrations were not really working recently and this change is introducing them back. (cherry picked from commit 68217f5)
) Integration tests so far were a separate test type among the unit tests, however we have to start them differently. This PR introduces new command in Breeze: breeze testing integration-tests The `--integration` option has been removed from the regular unit tests, and now it is used to the integration-tests command. The integration-tests command has no parallel option. (cherry picked from commit b37452e)
ephraimbuddy
force-pushed
the
v2-5-test
branch
from
2593275 to
2ce4b56
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Time for
2.5.1rc1!