Skip to content

Commit

Permalink
Move apache-airflow docs to subdirectory (apache#12715)
Browse files Browse the repository at this point in the history
  • Loading branch information
mik-laj authored Dec 1, 2020
1 parent dee304b commit 42f0a3d
Show file tree
Hide file tree
Showing 155 changed files with 64 additions and 94 deletions.
6 changes: 0 additions & 6 deletions docs/README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -22,12 +22,6 @@ This directory contains documentation for the Apache Airflow project and other p

For Helm Chart, see: `/chart/README.md <../chart/READMe.md>`__

File structure
==============

Currently in the ``/docs/`` directory there is documentation for the ``apache-airflow`` package. The ``apache-airflow-providers-*'` directories contains independent documentation for each provider package.
We are taking steps to move the documentation for ``apache-airflow`` package to the ``apache-airflow`` subdirectory to make the directory structure clearer.

Development documentation preview
==================================

Expand Down
File renamed without changes.
File renamed without changes.
2 changes: 1 addition & 1 deletion docs/changelog.rst → docs/apache-airflow/changelog.rst
Original file line number Diff line number Diff line change
Expand Up @@ -20,4 +20,4 @@
Changelog
=========

.. include:: ../CHANGELOG.txt
.. include:: ../../CHANGELOG.txt
File renamed without changes.
20 changes: 10 additions & 10 deletions docs/concepts.rst → docs/apache-airflow/concepts.rst
Original file line number Diff line number Diff line change
Expand Up @@ -183,7 +183,7 @@ You can also use the parameters on jinja templates by using the ``{{context.para

Example DAG with decorator:

.. exampleinclude:: /../airflow/example_dags/example_dag_decorator.py
.. exampleinclude:: /../../airflow/example_dags/example_dag_decorator.py
:language: python
:start-after: [START dag_decorator_usage]
:end-before: [END dag_decorator_usage]
Expand Down Expand Up @@ -939,14 +939,14 @@ Note that SubDAG operators should contain a factory method that returns a DAG
object. This will prevent the SubDAG from being treated like a separate DAG in
the main UI. For example:

.. exampleinclude:: /../airflow/example_dags/subdags/subdag.py
.. exampleinclude:: /../../airflow/example_dags/subdags/subdag.py
:language: python
:start-after: [START subdag]
:end-before: [END subdag]

This SubDAG can then be referenced in your main DAG file:

.. exampleinclude:: /../airflow/example_dags/example_subdag_operator.py
.. exampleinclude:: /../../airflow/example_dags/example_subdag_operator.py
:language: python
:start-after: [START example_subdag_operator]
:end-before: [END example_subdag_operator]
Expand Down Expand Up @@ -1013,7 +1013,7 @@ and then puts both tasks upstream of ``task3``:

Here is a more complicated example DAG with multiple levels of nested TaskGroups:

.. exampleinclude:: /../airflow/example_dags/example_task_group.py
.. exampleinclude:: /../../airflow/example_dags/example_task_group.py
:language: python
:start-after: [START howto_task_group]
:end-before: [END howto_task_group]
Expand Down Expand Up @@ -1158,7 +1158,7 @@ right now is not between its ``execution_time`` and the next scheduled

For example, consider the following DAG:

.. exampleinclude:: /../airflow/example_dags/example_latest_only_with_trigger.py
.. exampleinclude:: /../../airflow/example_dags/example_latest_only_with_trigger.py
:language: python
:start-after: [START example]
:end-before: [END example]
Expand Down Expand Up @@ -1240,7 +1240,7 @@ DAG level cluster policy
In this example we check if each DAG has at least one tag defined.
Here is what it may look like:

.. literalinclude:: /../tests/cluster_policies/__init__.py
.. literalinclude:: /../../tests/cluster_policies/__init__.py
:language: python
:start-after: [START example_dag_cluster_policy]
:end-before: [END example_dag_cluster_policy]
Expand All @@ -1252,7 +1252,7 @@ using a specific operator, or enforce a task timeout policy, making sure
that no tasks run for more than 48 hours. Here's an example of what this
may look like:

.. literalinclude:: /../tests/cluster_policies/__init__.py
.. literalinclude:: /../../tests/cluster_policies/__init__.py
:language: python
:start-after: [START example_task_cluster_policy]
:end-before: [END example_task_cluster_policy]
Expand All @@ -1263,7 +1263,7 @@ reviewer, rather than as technical security controls.

For example, don't run tasks without airflow owners:

.. literalinclude:: /../tests/cluster_policies/__init__.py
.. literalinclude:: /../../tests/cluster_policies/__init__.py
:language: python
:start-after: [START example_cluster_policy_rule]
:end-before: [END example_cluster_policy_rule]
Expand All @@ -1276,7 +1276,7 @@ the UI (and import errors table in the database).

For Example in ``airflow_local_settings.py``:

.. literalinclude:: /../tests/cluster_policies/__init__.py
.. literalinclude:: /../../tests/cluster_policies/__init__.py
:language: python
:start-after: [START example_list_of_cluster_policy_rules]
:end-before: [END example_list_of_cluster_policy_rules]
Expand All @@ -1286,7 +1286,7 @@ Task instance mutation hook
Task instance mutation hook can be used for example to re-routes the task to
execute in a different queue during retries:

.. literalinclude:: /../tests/cluster_policies/__init__.py
.. literalinclude:: /../../tests/cluster_policies/__init__.py
:language: python
:start-after: [START example_task_mutation_hook]
:end-before: [END example_task_mutation_hook]
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -77,21 +77,21 @@ With these requirements in mind, here are some examples of basic ``pod_template_

pod_template_file using the ``dag_in_image`` setting:

.. exampleinclude:: /../airflow/kubernetes/pod_template_file_examples/dags_in_image_template.yaml
.. exampleinclude:: /../../airflow/kubernetes/pod_template_file_examples/dags_in_image_template.yaml
:language: yaml
:start-after: [START template_with_dags_in_image]
:end-before: [END template_with_dags_in_image]

``pod_template_file`` which stores DAGs in a ``persistentVolume``:

.. exampleinclude:: /../airflow/kubernetes/pod_template_file_examples/dags_in_volume_template.yaml
.. exampleinclude:: /../../airflow/kubernetes/pod_template_file_examples/dags_in_volume_template.yaml
:language: yaml
:start-after: [START template_with_dags_in_volume]
:end-before: [END template_with_dags_in_volume]

``pod_template_file`` which pulls DAGs from git:

.. exampleinclude:: /../airflow/kubernetes/pod_template_file_examples/git_sync_template.yaml
.. exampleinclude:: /../../airflow/kubernetes/pod_template_file_examples/git_sync_template.yaml
:language: yaml
:start-after: [START git_sync_template]
:end-before: [END git_sync_template]
Expand All @@ -108,7 +108,7 @@ Please note that the scheduler will override the ``metadata.name`` of the V1pod
To overwrite the base container of the pod launched by the KubernetesExecutor,
create a V1pod with a single container, and overwrite the fields as follows:

.. exampleinclude:: /../airflow/example_dags/example_kubernetes_executor_config.py
.. exampleinclude:: /../../airflow/example_dags/example_kubernetes_executor_config.py
:language: python
:start-after: [START task_with_volume]
:end-before: [END task_with_volume]
Expand All @@ -118,7 +118,7 @@ Note that volume mounts environment variables, ports, and devices will all be ex
To add a sidecar container to the launched pod, create a V1pod with an empty first container with the
name ``base`` and a second container containing your desired sidecar.

.. exampleinclude:: /../airflow/example_dags/example_kubernetes_executor_config.py
.. exampleinclude:: /../../airflow/example_dags/example_kubernetes_executor_config.py
:language: python
:start-after: [START task_with_sidecar]
:end-before: [END task_with_sidecar]
Expand All @@ -128,7 +128,7 @@ This will replace the default ``pod_template_file`` named in the airflow.cfg and

Here is an example of a task with both features:

.. exampleinclude:: /../airflow/example_dags/example_kubernetes_executor_config.py
.. exampleinclude:: /../../airflow/example_dags/example_kubernetes_executor_config.py
:language: python
:start-after: [START task_with_template]
:end-before: [END task_with_template]
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ BashOperator
Use the :class:`~airflow.operators.bash.BashOperator` to execute
commands in a `Bash <https://www.gnu.org/software/bash/>`__ shell.

.. exampleinclude:: /../airflow/example_dags/example_bash_operator.py
.. exampleinclude:: /../../airflow/example_dags/example_bash_operator.py
:language: python
:start-after: [START howto_operator_bash]
:end-before: [END howto_operator_bash]
Expand All @@ -36,7 +36,7 @@ Templating
You can use :ref:`Jinja templates <jinja-templating>` to parameterize the
``bash_command`` argument.

.. exampleinclude:: /../airflow/example_dags/example_bash_operator.py
.. exampleinclude:: /../../airflow/example_dags/example_bash_operator.py
:language: python
:start-after: [START howto_operator_bash_template]
:end-before: [END howto_operator_bash_template]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ wait for another task on a different DAG for a specific ``execution_date``.
ExternalTaskSensor also provide options to set if the Task on a remote DAG succeeded or failed
via ``allowed_states`` and ``failed_states`` parameters.

.. exampleinclude:: /../airflow/example_dags/example_external_task_marker_dag.py
.. exampleinclude:: /../../airflow/example_dags/example_external_task_marker_dag.py
:language: python
:start-after: [START howto_operator_external_task_sensor]
:end-before: [END howto_operator_external_task_sensor]
Expand All @@ -58,7 +58,7 @@ on ``child_dag`` for a specific ``execution_date`` should also be cleared, ``Ext
should be used. Note that ``child_task1`` will only be cleared if "Recursive" is selected when the
user clears ``parent_task``.

.. exampleinclude:: /../airflow/example_dags/example_external_task_marker_dag.py
.. exampleinclude:: /../../airflow/example_dags/example_external_task_marker_dag.py
:language: python
:start-after: [START howto_operator_external_task_marker]
:end-before: [END howto_operator_external_task_marker]
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ determine what actually executes when your DAG runs.

.. note::
See the :ref:`Operators Concepts <concepts:operators>` documentation and the
:doc:`Operators API Reference <../../_api/index>` for more
:doc:`Operators API Reference </python-api-ref>` for more
information.

.. toctree::
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ PythonOperator
Use the :class:`~airflow.operators.python.PythonOperator` to execute
Python callables.

.. exampleinclude:: /../airflow/example_dags/example_python_operator.py
.. exampleinclude:: /../../airflow/example_dags/example_python_operator.py
:language: python
:start-after: [START howto_operator_python]
:end-before: [END howto_operator_python]
Expand All @@ -36,7 +36,7 @@ Passing in arguments
Use the ``op_args`` and ``op_kwargs`` arguments to pass additional arguments
to the Python callable.

.. exampleinclude:: /../airflow/example_dags/example_python_operator.py
.. exampleinclude:: /../../airflow/example_dags/example_python_operator.py
:language: python
:start-after: [START howto_operator_python_kwargs]
:end-before: [END howto_operator_python_kwargs]
Expand All @@ -61,7 +61,7 @@ PythonVirtualenvOperator
Use the :class:`~airflow.operators.python.PythonVirtualenvOperator` to execute
Python callables inside a new Python virtual environment.

.. exampleinclude:: ../../../airflow/example_dags/example_python_operator.py
.. exampleinclude:: /../../airflow/example_dags/example_python_operator.py
:language: python
:start-after: [START howto_operator_python_venv]
:end-before: [END howto_operator_python_venv]
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes.
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
4 changes: 2 additions & 2 deletions docs/index.rst → docs/apache-airflow/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
.. image:: ../airflow/www/static/pin_large.png
.. image:: ../../airflow/www/static/pin_large.png
:width: 100

Apache Airflow Documentation
Expand Down Expand Up @@ -112,7 +112,7 @@ Content
Operators and hooks <operators-and-hooks-ref>
CLI <cli-and-env-variables-ref>
Macros <macros-ref>
Python API <_api/index>
Python API <python-api-ref>
Experimental REST API <rest-api-ref>
Stable REST API <stable-rest-api-ref>
Configurations <configurations-ref>
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -62,9 +62,9 @@ All operators are in the following packages:
:glob:
:maxdepth: 1

airflow/operators/index
_api/airflow/operators/index

airflow/sensors/index
_api/airflow/sensors/index


.. _pythonapi:hooks:
Expand All @@ -84,7 +84,7 @@ All hooks are in the following packages:
:glob:
:maxdepth: 1

airflow/hooks/index
_api/airflow/hooks/index

Executors
---------
Expand All @@ -100,7 +100,7 @@ All executors are in the following packages:
:glob:
:maxdepth: 1

airflow/executors/index
_api/airflow/executors/index

Models
------
Expand All @@ -112,7 +112,7 @@ persisted in the database.
:glob:
:maxdepth: 1

airflow/models/index
_api/airflow/models/index

.. _pythonapi:exceptions:

Expand All @@ -124,7 +124,7 @@ Exceptions
:glob:
:maxdepth: 1

airflow/exceptions/index
_api/airflow/exceptions/index

Secrets Backends
----------------
Expand All @@ -136,4 +136,4 @@ All secrets backends derive from :class:`~airflow.secrets.BaseSecretsBackend`.
:glob:
:maxdepth: 1

airflow/secrets/index
_api/airflow/secrets/index
1 change: 1 addition & 0 deletions docs/redirects.txt → docs/apache-airflow/redirects.txt
Original file line number Diff line number Diff line change
Expand Up @@ -85,3 +85,4 @@ errors.rst logging-monitoring/errors.rst
howto/write-logs.rst logging-monitoring/logging-tasks.rst
metrics.rst logging-monitoring/metrics.rst
howto/tracking-user-activity.rst logging-monitoring/tracking-user-activity.rst
_api/index.rst python-api-ref.rst
File renamed without changes.
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ Viewer
^^^^^^
``Viewer`` users have limited viewer permissions

.. exampleinclude:: /../airflow/www/security.py
.. exampleinclude:: /../../airflow/www/security.py
:language: python
:start-after: [START security_viewer_perms]
:end-before: [END security_viewer_perms]
Expand All @@ -66,7 +66,7 @@ User
^^^^
``User`` users have ``Viewer`` permissions plus additional user permissions

.. exampleinclude:: /../airflow/www/security.py
.. exampleinclude:: /../../airflow/www/security.py
:language: python
:start-after: [START security_user_perms]
:end-before: [END security_user_perms]
Expand All @@ -77,7 +77,7 @@ Op
^^
``Op`` users have ``User`` permissions plus additional op permissions

.. exampleinclude:: /../airflow/www/security.py
.. exampleinclude:: /../../airflow/www/security.py
:language: python
:start-after: [START security_op_perms]
:end-before: [END security_op_perms]
Expand Down
File renamed without changes.
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -27,4 +27,4 @@ Security
*
secrets/index

.. include:: /../.github/SECURITY.rst
.. include:: /../../.github/SECURITY.rst
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
Loading

0 comments on commit 42f0a3d

Please sign in to comment.