Error: ModuleNotFoundError: No module named 'airflow' | Docker Image Windows WSL2

[anilkulkarni87]

Apache Airflow version: 2.0.0

Kubernetes version (if you are using kubernetes) (use kubectl version): NA

Environment: Local Windows Laptop

• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release): Windows 10 (WSL 2)
• Install tools: Docker desktop in Windows 10 (WSL2)
• Others:

What happened:

I am trying to setup local development with Airflow and was using the Docker-compose.yaml file in the repo. I am encountering the below error when I issues the command 'docker-compose up airflow-init'

Starting airflowdocker_redis_1 ... done
Starting airflowdocker_postgres_1 ... done
Starting airflowdocker_airflow-init_1 ... done
Attaching to airflowdocker_airflow-init_1
airflow-init_1 | DB_BACKEND=postgresql+psycopg2
airflow-init_1 | DB_HOST=postgres
airflow-init_1 | DB_PORT=5432
airflow-init_1 |
airflow-init_1 | Traceback (most recent call last):
airflow-init_1 | File "/home/airflow/.local/bin/airflow", line 5, in
airflow-init_1 | from airflow.main import main
airflow-init_1 | ModuleNotFoundError: No module named 'airflow'
airflowdocker_airflow-init_1 exited with code 1

What you expected to happen:
I expected the containers to start and login to airflow via localhost:8080

How to reproduce it:
Get the docker-compose.yaml from the repo. and then just follow the instructions on a windows machine.

Anything else we need to know:
This occurs everytime

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[turbaszek]

I had this problem yesterday - it was down to the user config. It was fixed by setting in docker-compose:

user: "${AIRFLOW_UID:-1000}:${AIRFLOW_GID:-0}"

[anilkulkarni87]

I had this problem yesterday - it was down to the user config. It was fixed by setting in docker-compose:

user: "${AIRFLOW_UID:-1000}:${AIRFLOW_GID:-0}"

@turbaszek I changed the user config but I still seem to get the same error. Is there any other change that you missed to mention?

[anilkulkarni87]

@turbaszek I was able to fix the error.
Solution : I reset my WSL installation and setup a new user when i restarted it.
But now the problem is docker-compose up airflow-init is not creating the Admin user. and there docker-compose up is failing.

airflowdocker_postgres_1 is up-to-date
airflowdocker_redis_1 is up-to-date
Recreating airflowdocker_airflow-init_1 ... done
Attaching to airflowdocker_airflow-init_1
airflow-init_1 | DB_BACKEND=postgresql+psycopg2
airflow-init_1 | DB_HOST=postgres
airflow-init_1 | DB_PORT=5432
airflow-init_1 |
airflow-init_1 | 2.0.0
airflowdocker_airflow-init_1 exited with code 0

[turbaszek]

@anilkulkarni87 can you share your docker-compose? CC @mik-laj who's the author of this docker-compose 🐈

[anilkulkarni87]

Here is the file. @turbaszek

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.
#

# Basic Airflow cluster configuration for CeleryExecutor with Redis and PostgreSQL.
#
# WARNING: This configuration is for local development. Do not use it in a production deployment.
#
# This configuration supports basic configuration using environment variables or an .env file
# The following variables are supported:
#
# AIRFLOW_IMAGE_NAME         - Docker image name used to run Airflow.
#                              Default: apache/airflow:master-python3.8
# AIRFLOW_UID                - User ID in Airflow containers
#                              Default: 50000
# AIRFLOW_GID                - Group ID in Airflow containers
#                              Default: 50000
# _AIRFLOW_WWW_USER_USERNAME - Username for the administrator account.
#                              Default: airflow
# _AIRFLOW_WWW_USER_PASSWORD - Password for the administrator account.
#                              Default: airflow
#
# Feel free to modify this file to suit your needs.
---
version: '3'
x-airflow-common:
  &airflow-common
  image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.0.0-python3.8}
  environment:
    &airflow-common-env
    AIRFLOW__CORE__EXECUTOR: CeleryExecutor
    AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
    AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:airflow@postgres/airflow
    AIRFLOW__CELERY__BROKER_URL: redis://:@redis:6379/0
    AIRFLOW__CORE__FERNET_KEY: ''
    AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
    AIRFLOW__CORE__LOAD_EXAMPLES: 'true'
  volumes:
    - ./dags:/opt/airflow/dags
    - ./logs:/opt/airflow/logs
    - ./plugins:/opt/airflow/plugins
  user: "${AIRFLOW_UID:-0}:${AIRFLOW_GID:-0}"
  depends_on:
    redis:
      condition: service_healthy
    postgres:
      condition: service_healthy

services:
  postgres:
    image: postgres:13
    environment:
      POSTGRES_USER: airflow
      POSTGRES_PASSWORD: airflow
      POSTGRES_DB: airflow
    volumes:
      - postgres-db-volume:/var/lib/postgresql/data
    ports:
      - "5433:5432"
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "airflow"]
      interval: 5s
      retries: 5
    restart: always

  redis:
    image: redis:latest
    ports:
      - 6379:6379
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 30s
      retries: 50
    restart: always

  airflow-webserver:
    <<: *airflow-common
    command: webserver
    ports:
      - 8080:8080
    healthcheck:
      test: ["CMD", "curl", "--fail", "http://localhost:8080/health"]
      interval: 10s
      timeout: 10s
      retries: 5
    restart: always

  airflow-scheduler:
    <<: *airflow-common
    command: scheduler
    restart: always

  airflow-worker:
    <<: *airflow-common
    command: celery worker
    restart: always

  airflow-init:
    <<: *airflow-common
    command: version
    environment:
      <<: *airflow-common-env
      _AIRFLOW_DB_UPGRADE: 'true'
      _AIRFLOW_WWW_USER_CREATE: 'true'
      _AIRFLOW_WWW_USER_USERNAME: ${_AIRFLOW_WWW_USER_USERNAME:-airflow}
      _AIRFLOW_WWW_USER_PASSWORD: ${_AIRFLOW_WWW_USER_PASSWORD:-airflow}

  flower:
    <<: *airflow-common
    command: celery flower
    ports:
      - 5555:5555
    healthcheck:
      test: ["CMD", "curl", "--fail", "http://localhost:5555/"]
      interval: 10s
      timeout: 10s
      retries: 5
    restart: always

volumes:
  postgres-db-volume:

[mik-laj]

@turbaszek Windows is not yet supported and this file has not been tested in any way on this system. Unfortunately docker-compose for Windows is very limited and buggy. Other projects provide virtual machine images as the recommended method for using their projects on Windows.

For example, Superset:

Windows

Superset is not officially supported on Windows unfortunately. The best option for Windows users to try out Superset locally is to install an Ubuntu Desktop VM via VirtualBox and proceed with the Docker on Linux instructions inside of that VM. We recommend assigning at least 8GB of RAM to the virtual machine as well as provisioning a hard drive of at least 40GB, so that there will be enough space for both the OS and all of the required dependencies.

https://superset.apache.org/docs/installation/installing-superset-using-docker-compose

We should probably describe what Windows support looks like and prepare workarounds. We even have a ticket about it:
#13838

I recently bought a few computers with Windows and when time permits I will experiment with it a bit.

[anilkulkarni87]

@mik-laj Thanks for the update. I am currently using the Ubuntu on WSL2 in Windows 10 home. I was able to get it working with some small tweaks to the original docker-compose.yaml and tried my best to make it simpler.
I was successful and also was able to run a test data.

I captured as much info I could and saved it in my repo.

Please review(the changes in the docker-compose.yaml and would be great if you could test this in other machines) and then maybe we could just close this. I am just happy that I got it working for now.

[mik-laj]

As for docker-compose.yaml, depending on the Airflow version, we have different files.
For Airflow 2.0.1.and newer: https://github.com/apache/airflow/blob/master/docs/apache-airflow/start/docker-compose.yaml
For Airflow 2.0.0: https://airflow.apache.org/docs/apache-airflow/2.0.0/docker-compose.yaml

The main difference is a different command in the airflow-init service.

[anilkulkarni87]

Yes i noticed that and accommodated that change. Would like you review it and probably test in another machine. Please let me know.

The airflow init through /bin/bash did not work for me. I had tweak that

[ephraimbuddy]

Experiencing the same issue in ubuntu. And for 2.0.1 compose file, the default image tag is not complete:

airflow/docs/apache-airflow/start/docker-compose.yaml

Line 42 in beb8af5

image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.0.1}

Should be airflow:2.0.1-python3.8

[anilkulkarni87]

@mik-laj Where can i find the code that does airflow-init? Setting _AIRFLOW_DB_UPGRADE: 'true' What does it do?

airflow-init:
  <<: *airflow-common
  command: version
  environment:
    <<: *airflow-common-env
    _AIRFLOW_DB_UPGRADE: 'true'
    _AIRFLOW_WWW_USER_CREATE: 'true'
    _AIRFLOW_WWW_USER_USERNAME: ${_AIRFLOW_WWW_USER_USERNAME:-airflow}
    _AIRFLOW_WWW_USER_PASSWORD: ${_AIRFLOW_WWW_USER_PASSWORD:-airflow}

[mik-laj]

@anilkulkarni87 see: https://github.com/apache/airflow/pull/13728/files
I am not a big fan of it, but such features was added so I started using it.

[anilkulkarni87]

@mik-laj I think we could close this. I just tried the latest docker-compose on windows (WSL2 - Ubuntu) It works as expected. No issues at all.

[potiuk]

Closing.

BTW. Those _* is evaluated and executed in entrypoint and it is pretty obscure feature indeed (hence _* convention). Actually there is no need to use it in the docker-compose of ours - we could as easily use commands to run.

The main reason it is there is for cases where we want to run the production image standalone and make sure that when you enter the image, the database is initialized - it's mostly for development and testing when sqlite database is used.

Since this is a separate command to run in docker compose, I think it would be ok to replace it with direct commands to run in the image. Feel free to change it @mik-laj