Skip to content

Commit

Permalink
[AIRFLOW-XXX] Fix development packages installtion instructions (#6942)
Browse files Browse the repository at this point in the history
  • Loading branch information
xinbinhuang authored and turbaszek committed Dec 31, 2019
1 parent 83ba8e6 commit da2a617
Show file tree
Hide file tree
Showing 2 changed files with 37 additions and 27 deletions.
60 changes: 35 additions & 25 deletions LOCAL_VIRTUALENV.rst
Original file line number Diff line number Diff line change
Expand Up @@ -91,26 +91,51 @@ To use your IDE for Airflow development and testing, you need to configure a vir
environment. Ideally you should set up virtualenv for all Python versions that Airflow
supports (3.5, 3.6).

Consider using one of the following utilities to create virtual environments and easily
switch between them with the ``workon`` command:
To create and initialize the local virtualenv:

- `pyenv <https://github.com/pyenv/pyenv>`_
- `pyenv-virtualenv <https://github.com/pyenv/pyenv-virtualenv>`_
- `virtualenvwrapper <https://virtualenvwrapper.readthedocs.io/en/latest/>`_
1. Create an environment with one of the two options:

To create and initialize the local virtualenv:
- Option 1: consider using one of the following utilities to create virtual environments and easily switch between them with the ``workon`` command:

- `pyenv <https://github.com/pyenv/pyenv>`_
- `pyenv-virtualenv <https://github.com/pyenv/pyenv-virtualenv>`_
- `virtualenvwrapper <https://virtualenvwrapper.readthedocs.io/en/latest/>`_

``mkvirtualenv <ENV_NAME> --python=python<VERSION>``

- Option 2: create a local virtualenv with Conda

1. Create an environment as follows:
- install `miniconda3 <https://docs.conda.io/en/latest/miniconda.html>`_

``mkvirtualenv <ENV_NAME> --python=python<VERSION>``
.. code-block:: bash
conda create -n airflow python=3.6
conda activate airflow
2. Install Python PIP requirements:

``pip install -e ".[devel]"``
.. code-block:: bash
pip install -U -e ".[devel,<OTHER EXTRAS>]" # for example: pip install -U -e ".[devel,gcp,postgres]"
Note: when you first initialize database (the next step), you may encounter some problems.
This is because airflow by default will try to load in example dags where some of them requires dependencies ``gcp`` and ``postgres``.
You can solve the problem by:

- installing the extras i.e. ``[devel,gcp,postgres]`` or
- disable the example dags with environment variable: ``export AIRFLOW__CORE__LOAD_EXAMPLES=False`` or
- simply ignore the error messages and proceed

*In addition to above, you may also encounter problems during database migration.*
*This is a known issue and please see the progress here:* `AIRFLOW-6265 <https://issues.apache.org/jira/browse/AIRFLOW-6265>`_

3. Create the Airflow sqlite database:

``airflow db init``
.. code-block:: bash
# if necessary, start with a clean AIRFLOW_HOME, e.g.
# rm -rf ~/airflow
airflow db init
4. Select the virtualenv you created as the project's default virtualenv in your IDE.

Expand All @@ -122,21 +147,6 @@ Simply enter the Breeze environment by using ``workon`` and, once you are in it,
./breeze --initialize-local-virtualenv
Optional - create a local virtualenv with conda
-----------------------------------------------

- install `miniconda3 <https://docs.conda.io/en/latest/miniconda.html>`_

.. code-block:: bash
conda create -n airflow python=3.6
conda activate airflow
pip install -U -e ".[devel,gcp,postgres,ssh]"
# if necessary, start with a clean AIRFLOW_HOME, e.g.
# rm -rf ~/airflow
airflow db init
Running Tests
-------------

Expand Down
4 changes: 2 additions & 2 deletions STATIC_CODE_CHECKS.rst
Original file line number Diff line number Diff line change
Expand Up @@ -133,9 +133,9 @@ them with ``sudo apt install``, on macOS - with ``brew install``.

The current list of prerequisites is limited to ``xmllint``:

- on Linux, install via ``sudo apt install xmllint``;
- on Linux, install via ``sudo apt install libxml2-utils``;

- on macOS, install via ``brew install xmllint``.
- on macOS, install via ``brew install libxml2``.

Enabling Pre-commit Hooks
.........................
Expand Down

0 comments on commit da2a617

Please sign in to comment.