From efc538db6409525a3805b4b61cf88908e603c974 Mon Sep 17 00:00:00 2001 From: Xinbin Huang Date: Mon, 30 Dec 2019 23:19:03 -0800 Subject: [PATCH] [AIRFLOW-XXX] Fix development packages installtion instructions (#6942) (cherry picked from commit da2a617cb7494c6e358b079a6cf8c4129e742a63) --- LOCAL_VIRTUALENV.rst | 60 ++++++++++++++++++++++++------------------ STATIC_CODE_CHECKS.rst | 4 +-- 2 files changed, 37 insertions(+), 27 deletions(-) diff --git a/LOCAL_VIRTUALENV.rst b/LOCAL_VIRTUALENV.rst index a1e98562dc01f..6d54d6f5fd4af 100644 --- a/LOCAL_VIRTUALENV.rst +++ b/LOCAL_VIRTUALENV.rst @@ -91,26 +91,51 @@ To use your IDE for Airflow development and testing, you need to configure a vir environment. Ideally you should set up virtualenv for all Python versions that Airflow supports (2.7, 3.5, 3.6). -Consider using one of the following utilities to create virtual environments and easily -switch between them with the ``workon`` command: +To create and initialize the local virtualenv: -- `pyenv `_ -- `pyenv-virtualenv `_ -- `virtualenvwrapper `_ +1. Create an environment with one of the two options: -To create and initialize the local virtualenv: + - Option 1: consider using one of the following utilities to create virtual environments and easily switch between them with the ``workon`` command: + + - `pyenv `_ + - `pyenv-virtualenv `_ + - `virtualenvwrapper `_ + + ``mkvirtualenv --python=python`` + + - Option 2: create a local virtualenv with Conda -1. Create an environment as follows: + - install `miniconda3 `_ - ``mkvirtualenv --python=python`` + .. code-block:: bash + + conda create -n airflow python=3.6 + conda activate airflow 2. Install Python PIP requirements: - ``pip install -e ".[devel]"`` + .. code-block:: bash + + pip install -U -e ".[devel,]" # for example: pip install -U -e ".[devel,gcp,postgres]" + +Note: when you first initialize database (the next step), you may encounter some problems. +This is because airflow by default will try to load in example dags where some of them requires dependencies ``gcp`` and ``postgres``. +You can solve the problem by: + +- installing the extras i.e. ``[devel,gcp,postgres]`` or +- disable the example dags with environment variable: ``export AIRFLOW__CORE__LOAD_EXAMPLES=False`` or +- simply ignore the error messages and proceed + +*In addition to above, you may also encounter problems during database migration.* +*This is a known issue and please see the progress here:* `AIRFLOW-6265 `_ 3. Create the Airflow sqlite database: - ``airflow db init`` + .. code-block:: bash + + # if necessary, start with a clean AIRFLOW_HOME, e.g. + # rm -rf ~/airflow + airflow db init 4. Select the virtualenv you created as the project's default virtualenv in your IDE. @@ -122,21 +147,6 @@ Simply enter the Breeze environment by using ``workon`` and, once you are in it, ./breeze --initialize-local-virtualenv -Optional - create a local virtualenv with conda ------------------------------------------------ - -- install `miniconda3 `_ - -.. code-block:: bash - - conda create -n airflow python=3.6 - conda activate airflow - pip install -U -e ".[devel,gcp,postgres,ssh]" - - # if necessary, start with a clean AIRFLOW_HOME, e.g. - # rm -rf ~/airflow - airflow db init - Running Tests ------------- diff --git a/STATIC_CODE_CHECKS.rst b/STATIC_CODE_CHECKS.rst index 26204b4a0c4f7..51401b620f4d6 100644 --- a/STATIC_CODE_CHECKS.rst +++ b/STATIC_CODE_CHECKS.rst @@ -127,9 +127,9 @@ them with ``sudo apt install``, on macOS - with ``brew install``. The current list of prerequisites is limited to ``xmllint``: -- on Linux, install via ``sudo apt install xmllint``; +- on Linux, install via ``sudo apt install libxml2-utils``; -- on macOS, install via ``brew install xmllint``. +- on macOS, install via ``brew install libxml2``. Enabling Pre-commit Hooks .........................