Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error in setting up the development environment following the docs for VS code #18849

Closed
2 tasks done
Bowrna opened this issue Oct 9, 2021 · 5 comments · Fixed by #22127
Closed
2 tasks done

Error in setting up the development environment following the docs for VS code #18849

Bowrna opened this issue Oct 9, 2021 · 5 comments · Fixed by #22127
Labels
kind:bug This is a clearly a bug kind:documentation

Comments

@Bowrna
Copy link
Contributor

Bowrna commented Oct 9, 2021

Describe the issue with documentation

https://github.com/apache/airflow/blob/main/CONTRIBUTORS_QUICK_START.rst#setup-and-develop-using-visual-studio-code

I am following the steps on the above page to set up the development environment. however, running the
pyenv install 3.8.5 in Apple M1 chip MacOS Big Sur runs into an error similar to the issue mentioned in the below StackOverflow link

https://stackoverflow.com/questions/65457674/unexpected-output-of-arch-on-osx-using-mac-m1-installing-elastic-beans

How to solve the problem

I am not sure how to solve this issue, but i used higher version of python 3.8.10 that is supported in M1 chip Apple MacOS Big Sur to solve the issue for time being.

Anything else

Further following the steps in this documentation, i got docker container up and running. however, if i try to reset the db, i get error saying

sqlalchemy.exc.OperationalError: (MySQLdb._exceptions.OperationalError) (1130, "Host 'docker-compose_airflow_run_718f404eac1d.docker-compose_defau' is not allowed to connect to this MySQL server")(Background on this error at: http://sqlalche.me/e/13/e3q8)

I am not sure why if i am running into this issue.

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@Bowrna Bowrna added kind:bug This is a clearly a bug kind:documentation labels Oct 9, 2021
@boring-cyborg
Copy link

boring-cyborg bot commented Oct 9, 2021

Thanks for opening your first issue here! Be sure to follow the issue template!

@potiuk
Copy link
Member

potiuk commented Oct 9, 2021

M1 has some issues indeed. I believe the problem might be that MySQL still does not have ARM image.

@potiuk
Copy link
Member

potiuk commented Oct 9, 2021

But If you managed to fix or at least improve some of those descriptions - feel free!

@Bowrna
Copy link
Contributor Author

Bowrna commented Oct 11, 2021

Sure @potiuk I will add some improvements.

One more thing I forgot to add to this issue:
To get the docker container up, I used the following command
./breeze

Following is the CI image information when running above command:


                               Branch name:            main
                               Docker image:           ghcr.io/apache/airflow/main/ci/python3.8
                               Airflow source version: 2.2.0.dev0
                               Python version:         3.8
                               Backend:                mysql 5.7

However i got the below message when running ./breeze flags

****************************************************************************************************
 Choose backend to run for Airflow

-b, --backend BACKEND
        Backend to use for tests - it determines which database is used.
        One of:

               sqlite mysql postgres mssql

        Default: sqlite

I expected ./breeze to start with sqlite by default but it starts with mysql. Am I missing something?

@potiuk
Copy link
Member

potiuk commented Oct 11, 2021

I expected ./breeze to start with sqlite by default but it starts with mysql. Am I missing something?

Some of the parameters (like backend) are "saved" from the previous run and persisted (so that you can run "last used" configuration with just ./breeze.

It is described here https://github.com/apache/airflow/blob/main/BREEZE.rst#choosing-different-breeze-environment-configuration

You can use additional breeze flags to choose your environment. You can specify a Python version to use, and backend (the meta-data database). Thanks to that, with Breeze, you can recreate the same environments as we have in matrix builds in the CI.

For example, you can choose to run Python 3.6 tests with MySQL as backend and in the Docker environment as follows:

./breeze --python 3.6 --backend mysql
The choices you make are persisted in the ./.build/ cache directory so that next time when you use the breeze script, it could use the values that were used previously. This way you do not have to specify them when you run the script. You can delete the .build/ directory in case you want to restore the default settings.

potiuk added a commit to potiuk/airflow that referenced this issue Mar 10, 2022
This support is mostly for the developers, not for CI full chain yet.
It has several limitations:

* no MySQL client support
* no MsSQL client support
* no CI tests yet

What is implemented:

* automated detection of ARM/AMD architecture when building and
  running breeze
* automated cache refresh on CI for ARM/AMD

Currently only development (ghcr.io) images are supported for ARM.

Fixes: apache#18849
Fixes: apache#17494
Relates to: apache#15635

The images published in DockerHub for now are AMD64 only. We will
run development with M1 images for some time and later we will
likely make our DockerHub images multi-platform as well.

Also Hadolint does not have ARM images yet so we had to disable it
and we should re-enable it back after the support is added.
See hadolint/hadolint#411
potiuk added a commit that referenced this issue Mar 10, 2022
This support is mostly for the developers, not for CI full chain yet.
It has several limitations:

* no MySQL client support
* no MsSQL client support
* no CI tests yet

What is implemented:

* automated detection of ARM/AMD architecture when building and
  running breeze
* automated cache refresh on CI for ARM/AMD

Currently only development (ghcr.io) images are supported for ARM.

Fixes: #18849
Fixes: #17494
Relates to: #15635

The images published in DockerHub for now are AMD64 only. We will
run development with M1 images for some time and later we will
likely make our DockerHub images multi-platform as well.

Also Hadolint does not have ARM images yet so we had to disable it
and we should re-enable it back after the support is added.
See hadolint/hadolint#411
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind:bug This is a clearly a bug kind:documentation
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants