Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[AIRFLOW-XXXX] Fix development packages installtion instructions #6942

Merged
merged 1 commit into from
Dec 31, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 35 additions & 25 deletions LOCAL_VIRTUALENV.rst
Original file line number Diff line number Diff line change
Expand Up @@ -91,26 +91,51 @@ To use your IDE for Airflow development and testing, you need to configure a vir
environment. Ideally you should set up virtualenv for all Python versions that Airflow
supports (3.5, 3.6).

Consider using one of the following utilities to create virtual environments and easily
switch between them with the ``workon`` command:
To create and initialize the local virtualenv:

- `pyenv <https://github.com/pyenv/pyenv>`_
- `pyenv-virtualenv <https://github.com/pyenv/pyenv-virtualenv>`_
- `virtualenvwrapper <https://virtualenvwrapper.readthedocs.io/en/latest/>`_
1. Create an environment with one of the two options:

To create and initialize the local virtualenv:
- Option 1: consider using one of the following utilities to create virtual environments and easily switch between them with the ``workon`` command:

- `pyenv <https://github.com/pyenv/pyenv>`_
- `pyenv-virtualenv <https://github.com/pyenv/pyenv-virtualenv>`_
- `virtualenvwrapper <https://virtualenvwrapper.readthedocs.io/en/latest/>`_

``mkvirtualenv <ENV_NAME> --python=python<VERSION>``

- Option 2: create a local virtualenv with Conda

1. Create an environment as follows:
- install `miniconda3 <https://docs.conda.io/en/latest/miniconda.html>`_

``mkvirtualenv <ENV_NAME> --python=python<VERSION>``
.. code-block:: bash

conda create -n airflow python=3.6
conda activate airflow

2. Install Python PIP requirements:

``pip install -e ".[devel]"``
.. code-block:: bash

pip install -U -e ".[devel,<OTHER EXTRAS>]" # for example: pip install -U -e ".[devel,gcp,postgres]"

Note: when you first initialize database (the next step), you may encounter some problems.
This is because airflow by default will try to load in example dags where some of them requires dependencies ``gcp`` and ``postgres``.
You can solve the problem by:

- installing the extras i.e. ``[devel,gcp,postgres]`` or
- disable the example dags with environment variable: ``export AIRFLOW__CORE__LOAD_EXAMPLES=False`` or
- simply ignore the error messages and proceed
xinbinhuang marked this conversation as resolved.
Show resolved Hide resolved

*In addition to above, you may also encounter problems during database migration.*
*This is a known issue and please see the progress here:* `AIRFLOW-6265 <https://issues.apache.org/jira/browse/AIRFLOW-6265>`_

3. Create the Airflow sqlite database:

``airflow db init``
.. code-block:: bash

# if necessary, start with a clean AIRFLOW_HOME, e.g.
# rm -rf ~/airflow
airflow db init

4. Select the virtualenv you created as the project's default virtualenv in your IDE.

Expand All @@ -122,21 +147,6 @@ Simply enter the Breeze environment by using ``workon`` and, once you are in it,

./breeze --initialize-local-virtualenv

Optional - create a local virtualenv with conda
-----------------------------------------------

- install `miniconda3 <https://docs.conda.io/en/latest/miniconda.html>`_

.. code-block:: bash

conda create -n airflow python=3.6
conda activate airflow
pip install -U -e ".[devel,gcp,postgres,ssh]"

# if necessary, start with a clean AIRFLOW_HOME, e.g.
# rm -rf ~/airflow
airflow db init
xinbinhuang marked this conversation as resolved.
Show resolved Hide resolved

Running Tests
-------------

Expand Down
4 changes: 2 additions & 2 deletions STATIC_CODE_CHECKS.rst
Original file line number Diff line number Diff line change
Expand Up @@ -133,9 +133,9 @@ them with ``sudo apt install``, on macOS - with ``brew install``.

The current list of prerequisites is limited to ``xmllint``:

- on Linux, install via ``sudo apt install xmllint``;
- on Linux, install via ``sudo apt install libxml2-utils``;

- on macOS, install via ``brew install xmllint``.
- on macOS, install via ``brew install libxml2``.

Enabling Pre-commit Hooks
.........................
Expand Down