apache 2.5.1 -> affirm 2.5.1 #29
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…`RedshiftPauseClusterOperator` (apache#27276) * Modify RedshiftPauseClusterOperator and RedshiftResumeClusterOperator to attempt to pause and resume multiple times to avoid edge cases of state changes
It's been around a long, long time, but that doesn't make it any less confusing. I think it's time to do away with the "SNAP. Time is OUT" message, and replace it with something less cute / more clear / direct.
When one encounters "deadlock" in the logs, it's confusing because most people will immediately think database deadlock. But it's really about task dependencies. I thought about changing it to just "Dag has no runnable tasks", but the deadlong naming kindof has a meaning in the codebase so I tried to bridge the gap by clarifying it's specifically a _task_ deadlock.
* Allow datasets to be used in taskflow Datasets could not be passed as parameters to taskflow functions as they could not be serialized. This commit: 1) changes the xcom serializer so that it now can serialize objects that have attr, dataclass or a custom serializer 2) removes the need to for a custom serializer in lineage 3) adds a version check to the serializer/deserializer 4) registers any datasets as either inlets or outlets in the task 5) inlets or outlets that cannot be serialized now raise an error * Update docs/apache-airflow/concepts/taskflow.rst Co-authored-by: Bas Harenslak <[email protected]> * Apply suggestions from code review Co-authored-by: Bas Harenslak <[email protected]> Co-authored-by: Ash Berlin-Taylor <[email protected]>
PR 27540 left some formattng issues which weren't caught
* Make custom env vars optional for job tempaltes This addresses issue apache#26045, where an additional field is added to the job to determine whether or not any extra env fields are added to the job specification. This helps remove the issue currently with using CI/CD platforms like Argo that attempt to apply the env overrides to the job templates post their initial deploy which are immutable. Apply feedback from failing test
…apache#27754) This chnage limts the matrix of tests run on CI when full tests are run for all our tests - in the way that each of the items is for sure present but not repeated when unnecessary.
…tion) (apache#24079) * add audit log for AirflowModelViews this includes connections, variables, pools, sla, xcom. pools
Initially selective check was implemented in the way that it printed diagnostic output on stdout and the GITHUB_OUTPUT compatible set of outputs on stderr so that it could be redirected to the GITHUB_OUTPUT in its entirety. But this turned out to be a bad idea because when there was an error generated in selective-checks themselves, the traceback was printed in stderr and redirecting stderr to GITHUB_OUTPUT swallowed the traceback. This change reverses the behaviour: * diagnostic output is printed to stderr * GITHUB_OUTPUT compatible output is printed to stdout This way when traceback happens it is printed to stderr and is not swalleowed by redirection to GITHUB_OUTPUT (cherry picked from commit bc7feda)
…pache#28449) (cherry picked from commit 7950fb9)
to support both https/ssh (cherry picked from commit 7f9727f)
The recent isort changed their mind on sorting the imports. This change follows the change and bumps isort to latest released version (isort has no install_requires on its own so bumping min version has no effect on other dependencies) This change adds a number of isort:skip_file, isort:off, isort:skips in order to handle a very annoying bug in isort, that no matter how much you try, it sometimes treat "known first party" packages differently - depending on how many files it processes at a time. We should be able to restore it after this bug is fixed: PyCQA/isort#2045 This change also updates the common.sql API to skip them from isort for the very same reason (depending on how many files are modified, the isort order might change. (cherry picked from commit f115b20)
A big restructuring of our docs to make them easier to reason about - in a much more logical and consistent way. Likel other updates and content changes will be needed to make it even better. (cherry picked from commit 401fc57)
This PR performs housekeeping of the plugin examples: * makes the examples independent of Hive being installed * adds "has_access" in the examples * removes the misleading "metastore" (which is hive metastore not Airflow Metastore as used in other places This way our example will be much easier to apply by anyone. (cherry picked from commit 66eb282)
…8450) * Remove extra H1 & improve formatting of Listeners docs page I noticed that the documentation has an unclickable "Usage" page in the TOC. A little digging later, I discovered that this page contains an extra H1, and since this page is in the top level of the TOC, all the H1s on this page show up in the left docs sidebar. Demoted the "Usage" section to an H2, and fixed the other headers on this page to use consistent underlining with most other docs pages in this repo. I also took the liberty of sprucing up the language on the page to follow docs best practices, like shorter, highly readable sentences, title case in section titles, and bulleted lists to draw attention to important collections. * Remove extra newlines from specification discussion * Remove single newlines from listener API discussion * Remove nonexistent DagRun events from listeners page (cherry picked from commit 672264b)
…he#28650) The description is more clear now what Dynamic DAG generation is vs. Dynamic Task Mapping and note is added to the users to pay attention about the stable sorting that should be applied when generating DAGS. Related: apache#27523 (cherry picked from commit 36d887b)
…nd (apache#28261) The --integration tag should be standard flag rather than left for the common options. This change moves the flag to the right group. Images were regenerated because Rich does not know that the commands changed just when the option was moved to another group (this is a rich-click configuration and we are generating hash of commands from rich's command definition. As result of it, some of the breeze's svg files are changed. There are still subtle differences (mainly about font specification) on Linux and Maci and possibly it depends on what fonts are installed on your system - so when you regenerate images, font definition changes. This should be no issue in general as those images have the same hash and for all practical purposes, they are unchanged. (cherry picked from commit d8a0658)
Despite earlier attempts to fix the links there were still many places where the links were pointing to wrong versions of examples. It has been Noticed in: apache#27774 (comment) This change addresses it in three ways: * updated tool to fix the linx in historically released documentation (applied already in apache/airflow-site#706) * replaced hard-coded version with `|version|` everywhere and added sphinx extension to convert those to actual provider versions (though due to sphinx limitation it has to be done with post-processing of the generated .html rather than replacing during generation) * added pre-commit that detects any use of example_dags links to main/master/hard-coded version. (cherry picked from commit c8e348d)
…pache#28673) The newly released Pygments 2.14.0 changed slightly the way it wraps tags around rendeered variables and tour test made far too many assumptions about the rendered output which started to fail on main with the new Pygments. This PR fixes it by actually testing only what it was supposed to test: - lack of the secret in rendered output - presence of *** masking in the output (cherry picked from commit 81cd6c7)
(cherry picked from commit 49e178e)
The early cache is only needed when we push to main because it is only needed for PRs to the main brach. (cherry picked from commit 14783c6)
…inv (apache#28834) (cherry picked from commit 44d93be)
(cherry picked from commit 04306f1)
(cherry picked from commit e0bb6be)
…che#27696) (cherry picked from commit 518fd84)
This is the first stage of improving the way how integration tests are run in our CI - first we want to separate them in a separate packages, and then we want to run them separately - one integration each in the CI. (cherry picked from commit df608fe)
Previously we had separate integrations per service, but since we are moving the integration tests into separate job, it will be easier if the celery integration is a single one - this way we will have 1-1 relationship between tests to run and enabled integrations. The checks for various integrations were not really working recently and this change is introducing them back. (cherry picked from commit 68217f5)
…che#28207) Integration tests so far were a separate test type among the unit tests, however we have to start them differently. This PR introduces new command in Breeze: breeze testing integration-tests The `--integration` option has been removed from the regular unit tests, and now it is used to the integration-tests command. The integration-tests command has no parallel option. (cherry picked from commit b37452e)
…pache#28962) The dnspython has been released 2 hours ago and apparently eventlet that cassandra-driver uses is not compatible with the new version. Until the problem is fixed, we should limit dnspython to <2.3.0. Related: eventlet/eventlet#781 (cherry picked from commit d169df7)
The eventlet/eventlet#782 fixed error that caused the need to limit dnspython and eventlet 0.33.3 has been released. We can now remove the limit for dnspython as long as we also limit eventlet to >=0.33.3. (cherry picked from commit eaf5675)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.