Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add missing Python 3.11 dependencies. #29609

Closed
wants to merge 3 commits into from
Closed

Add missing Python 3.11 dependencies. #29609

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
8178190
Add missing Python 3.11 dependencies.
tvalentyn Dec 5, 2023
6891007
Also install headers in the python dependency test suite.
tvalentyn Dec 5, 2023
e6627d2
Add snappy instructions for MacOS
tvalentyn Dec 5, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/python_dependency_tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,8 @@ jobs:
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.params.py_ver }}
- name: Install headers required by some python packages
Copy link
Contributor

@AnandInguva AnandInguva Dec 5, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we might need to install this in the tox tests? for tests that might be using snappy? I don't think we install it tox anyway

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, it is a gap in beam test coverage. this codepath might be covered by google-internal tests that use TFRecordIo.

run: sudo apt-get install -y libsnappy-dev
- name: Install base_image_requirements.txt
working-directory: ./sdks/python
run: pip install --no-deps -r container/${{ matrix.params.py_env }}/base_image_requirements.txt
Expand Down
12 changes: 7 additions & 5 deletions .github/workflows/update_python_dependencies.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
# under the License.

# This workflow will update python dependencies as part of the release process
# And commit the changes into the branch release, creating a PR into the branch
# And commit the changes into the branch release, creating a PR into the branch

name: Update Python Depedencies

Expand All @@ -38,7 +38,7 @@ jobs:
properties: ${{ steps.test-properties.outputs.properties }}
steps:
- name: Checkout code
uses: actions/checkout@v4
uses: actions/checkout@v4
- id: test-properties
uses: ./.github/actions/setup-default-test-properties

Expand All @@ -57,6 +57,8 @@ jobs:
3.9
java-version: 8
go-version: 1.21
- name: Install headers required by some python packages
run: sudo apt-get install -y libsnappy-dev
- name: Update Python Dependencies
uses: ./.github/actions/gradle-command-self-hosted-action
with:
Expand All @@ -71,8 +73,8 @@ jobs:
run: |
branchName=weekly_update_python_dependencies_$(date +%s)
git checkout -b $branchName
git add -A
git diff-index --quiet HEAD || gitdiff=$? || echo $?
git add -A
git diff-index --quiet HEAD || gitdiff=$? || echo $?
if [[ $gitDiff != 0 ]]; then
echo "Changes are ready to commit"
git commit -m "Update Python Dependencies" --quiet
Expand All @@ -81,4 +83,4 @@ jobs:
echo "Link of the new PR $GITHUB_PR_URL"
else
echo "No changes on the files"
fi
fi
9 changes: 3 additions & 6 deletions sdks/python/container/base_image_requirements_manual.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,15 +32,12 @@ cython<1
# some versions of libraries that launch Beam pipelines, like tensorflow-transform.
# Leaving 'future' in our containers for now prevent breaking tft users.
future
# TODO(https://github.com/GoogleCloudPlatform/cloud-profiler-python/issues/127): Unpin python version.
google-cloud-profiler;python_version<"3.11"
google-cloud-profiler
guppy3
mmh3 # Optimizes execution of some Beam codepaths. TODO: Make it Beam's dependency.
nltk # Commonly used for natural language processing.
nose==1.3.7 # For Dataflow internal testing. TODO: remove this.
# TODO: Uncomment python version once python-snappy supports it.
# https://github.com/apache/beam/issues/25985
python-snappy;python_version<"3.11" # Optimizes execution of some Beam codepaths.
python-snappy # Optimizes execution of some Beam codepaths.
scipy
scikit-learn
build>=1.0,<2 # tool to build sdist from setup.py in stager.
build>=1.0,<2 # tool to build sdist from setup.py in stager.
20 changes: 10 additions & 10 deletions sdks/python/container/py310/base_image_requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ charset-normalizer==3.3.2
click==8.1.7
cloudpickle==2.2.1
crcmod==1.7
cryptography==41.0.6
cryptography==41.0.7
Cython==0.29.36
deprecation==2.1.0
dill==0.3.1.1
Expand All @@ -43,16 +43,16 @@ exceptiongroup==1.2.0
execnet==2.0.2
fastavro==1.9.0
fasteners==0.19
freezegun==1.2.2
freezegun==1.3.1
future==0.18.3
google-api-core==2.14.0
google-api-python-client==2.108.0
google-api-python-client==2.109.0
google-apitools==0.5.31
google-auth==2.23.4
google-auth==2.24.0
google-auth-httplib2==0.1.1
google-cloud-aiplatform==1.36.4
google-cloud-bigquery==3.13.0
google-cloud-bigquery-storage==2.22.0
google-cloud-bigquery-storage==2.23.0
google-cloud-bigtable==2.21.0
google-cloud-core==2.3.3
google-cloud-datastore==2.18.0
Expand Down Expand Up @@ -83,7 +83,7 @@ iniconfig==2.0.0
joblib==1.3.2
Js2Py==0.74
jsonschema==4.20.0
jsonschema-specifications==2023.11.1
jsonschema-specifications==2023.11.2
mmh3==4.0.1
mock==5.1.0
nltk==3.8.1
Expand All @@ -108,7 +108,7 @@ pycparser==2.21
pydot==1.4.2
PyHamcrest==2.1.0
pyjsparser==2.7.1
pymongo==4.6.0
pymongo==4.6.1
PyMySQL==1.1.0
pyparsing==3.1.1
pyproject_hooks==1.0.0
Expand All @@ -119,11 +119,11 @@ python-dateutil==2.8.2
python-snappy==0.6.1
pytz==2023.3.post1
PyYAML==6.0.1
referencing==0.31.0
referencing==0.31.1
regex==2023.10.3
requests==2.31.0
requests-mock==1.11.0
rpds-py==0.13.1
rpds-py==0.13.2
rsa==4.9
scikit-learn==1.3.2
scipy==1.11.4
Expand All @@ -142,6 +142,6 @@ typing_extensions==4.8.0
tzlocal==5.2
uritemplate==4.1.1
urllib3==2.1.0
websocket-client==1.6.4
websocket-client==1.7.0
wrapt==1.16.0
zstandard==0.22.0
22 changes: 13 additions & 9 deletions sdks/python/container/py311/base_image_requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ charset-normalizer==3.3.2
click==8.1.7
cloudpickle==2.2.1
crcmod==1.7
cryptography==41.0.6
cryptography==41.0.7
Cython==0.29.36
deprecation==2.1.0
dill==0.3.1.1
Expand All @@ -42,20 +42,22 @@ docopt==0.6.2
execnet==2.0.2
fastavro==1.9.0
fasteners==0.19
freezegun==1.2.2
freezegun==1.3.1
future==0.18.3
google-api-core==2.14.0
google-api-python-client==2.109.0
google-apitools==0.5.31
google-auth==2.23.4
google-auth==2.24.0
google-auth-httplib2==0.1.1
google-cloud-aiplatform==1.36.4
google-cloud-bigquery==3.13.0
google-cloud-bigquery-storage==2.22.0
google-cloud-bigquery-storage==2.23.0
google-cloud-bigtable==2.21.0
google-cloud-core==2.3.3
google-cloud-datastore==2.18.0
google-cloud-dlp==3.13.0
google-cloud-language==2.11.1
google-cloud-profiler==4.1.0
google-cloud-pubsub==2.18.4
google-cloud-pubsublite==1.8.3
google-cloud-recommendations-ai==0.10.5
Expand All @@ -80,7 +82,7 @@ iniconfig==2.0.0
joblib==1.3.2
Js2Py==0.74
jsonschema==4.20.0
jsonschema-specifications==2023.11.1
jsonschema-specifications==2023.11.2
mmh3==4.0.1
mock==5.1.0
nltk==3.8.1
Expand All @@ -105,21 +107,22 @@ pycparser==2.21
pydot==1.4.2
PyHamcrest==2.1.0
pyjsparser==2.7.1
pymongo==4.6.0
pymongo==4.6.1
PyMySQL==1.1.0
pyparsing==3.1.1
pyproject_hooks==1.0.0
pytest==7.4.3
pytest-timeout==2.2.0
pytest-xdist==3.5.0
python-dateutil==2.8.2
python-snappy==0.6.1
pytz==2023.3.post1
PyYAML==6.0.1
referencing==0.31.0
referencing==0.31.1
regex==2023.10.3
requests==2.31.0
requests-mock==1.11.0
rpds-py==0.13.1
rpds-py==0.13.2
rsa==4.9
scikit-learn==1.3.2
scipy==1.11.4
Expand All @@ -135,7 +138,8 @@ threadpoolctl==3.2.0
tqdm==4.66.1
typing_extensions==4.8.0
tzlocal==5.2
uritemplate==4.1.1
urllib3==2.1.0
websocket-client==1.6.4
websocket-client==1.7.0
wrapt==1.16.0
zstandard==0.22.0
22 changes: 11 additions & 11 deletions sdks/python/container/py38/base_image_requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ charset-normalizer==3.3.2
click==8.1.7
cloudpickle==2.2.1
crcmod==1.7
cryptography==41.0.6
cryptography==41.0.7
Cython==0.29.36
deprecation==2.1.0
dill==0.3.1.1
Expand All @@ -44,16 +44,16 @@ exceptiongroup==1.2.0
execnet==2.0.2
fastavro==1.9.0
fasteners==0.19
freezegun==1.2.2
freezegun==1.3.1
future==0.18.3
google-api-core==2.14.0
google-api-python-client==2.108.0
google-api-python-client==2.109.0
google-apitools==0.5.31
google-auth==2.23.4
google-auth==2.24.0
google-auth-httplib2==0.1.1
google-cloud-aiplatform==1.36.4
google-cloud-bigquery==3.13.0
google-cloud-bigquery-storage==2.22.0
google-cloud-bigquery-storage==2.23.0
google-cloud-bigtable==2.21.0
google-cloud-core==2.3.3
google-cloud-datastore==2.18.0
Expand All @@ -80,13 +80,13 @@ hdfs==2.7.3
httplib2==0.22.0
hypothesis==6.91.0
idna==3.6
importlib-metadata==6.8.0
importlib-metadata==7.0.0
importlib-resources==6.1.1
iniconfig==2.0.0
joblib==1.3.2
Js2Py==0.74
jsonschema==4.20.0
jsonschema-specifications==2023.11.1
jsonschema-specifications==2023.11.2
mmh3==4.0.1
mock==5.1.0
nltk==3.8.1
Expand All @@ -112,7 +112,7 @@ pycparser==2.21
pydot==1.4.2
PyHamcrest==2.1.0
pyjsparser==2.7.1
pymongo==4.6.0
pymongo==4.6.1
PyMySQL==1.1.0
pyparsing==3.1.1
pyproject_hooks==1.0.0
Expand All @@ -123,11 +123,11 @@ python-dateutil==2.8.2
python-snappy==0.6.1
pytz==2023.3.post1
PyYAML==6.0.1
referencing==0.31.0
referencing==0.31.1
regex==2023.10.3
requests==2.31.0
requests-mock==1.11.0
rpds-py==0.13.1
rpds-py==0.13.2
rsa==4.9
scikit-learn==1.3.2
scipy==1.10.1
Expand All @@ -146,7 +146,7 @@ typing_extensions==4.8.0
tzlocal==5.2
uritemplate==4.1.1
urllib3==2.1.0
websocket-client==1.6.4
websocket-client==1.7.0
wrapt==1.16.0
zipp==3.17.0
zstandard==0.22.0
22 changes: 11 additions & 11 deletions sdks/python/container/py39/base_image_requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ charset-normalizer==3.3.2
click==8.1.7
cloudpickle==2.2.1
crcmod==1.7
cryptography==41.0.6
cryptography==41.0.7
Cython==0.29.36
deprecation==2.1.0
dill==0.3.1.1
Expand All @@ -43,16 +43,16 @@ exceptiongroup==1.2.0
execnet==2.0.2
fastavro==1.9.0
fasteners==0.19
freezegun==1.2.2
freezegun==1.3.1
future==0.18.3
google-api-core==2.14.0
google-api-python-client==2.108.0
google-api-python-client==2.109.0
google-apitools==0.5.31
google-auth==2.23.4
google-auth==2.24.0
google-auth-httplib2==0.1.1
google-cloud-aiplatform==1.36.4
google-cloud-bigquery==3.13.0
google-cloud-bigquery-storage==2.22.0
google-cloud-bigquery-storage==2.23.0
google-cloud-bigtable==2.21.0
google-cloud-core==2.3.3
google-cloud-datastore==2.18.0
Expand All @@ -79,12 +79,12 @@ hdfs==2.7.3
httplib2==0.22.0
hypothesis==6.91.0
idna==3.6
importlib-metadata==6.8.0
importlib-metadata==7.0.0
iniconfig==2.0.0
joblib==1.3.2
Js2Py==0.74
jsonschema==4.20.0
jsonschema-specifications==2023.11.1
jsonschema-specifications==2023.11.2
mmh3==4.0.1
mock==5.1.0
nltk==3.8.1
Expand All @@ -109,7 +109,7 @@ pycparser==2.21
pydot==1.4.2
PyHamcrest==2.1.0
pyjsparser==2.7.1
pymongo==4.6.0
pymongo==4.6.1
PyMySQL==1.1.0
pyparsing==3.1.1
pyproject_hooks==1.0.0
Expand All @@ -120,11 +120,11 @@ python-dateutil==2.8.2
python-snappy==0.6.1
pytz==2023.3.post1
PyYAML==6.0.1
referencing==0.31.0
referencing==0.31.1
regex==2023.10.3
requests==2.31.0
requests-mock==1.11.0
rpds-py==0.13.1
rpds-py==0.13.2
rsa==4.9
scikit-learn==1.3.2
scipy==1.11.4
Expand All @@ -143,7 +143,7 @@ typing_extensions==4.8.0
tzlocal==5.2
uritemplate==4.1.1
urllib3==2.1.0
websocket-client==1.6.4
websocket-client==1.7.0
wrapt==1.16.0
zipp==3.17.0
zstandard==0.22.0
10 changes: 10 additions & 0 deletions sdks/python/container/run_generate_requirements.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,16 @@
# You will need Python interpreters for all versions supported by Beam, see:
# https://s.apache.org/beam-python-dev-wiki

if ! [[ "$(expr substr $(uname -s) 1 5)" == "Linux" ]]; then
echo This script needs to be executed in a Linux environment.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

optional: update wiki page since devs mostly update it on their own systems (which is Mac).

Copy link
Contributor Author

@tvalentyn tvalentyn Dec 5, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe the script already fails on Mac due to google-cloud-profiler or some other dependency not installable on Mac, that is why I added this. Can current version of the script (before this PR) run on Mac?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe Wiki already recommends a linux machine.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe Wiki already recommends a linux machine.

looks like not. I definitely remember script failing on MacOS before.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've run this successfully from my (non-M1) mac before

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, added instructions for MacOS. @AnandInguva please help verify that is sufficient. You can check out the pr locally by smth like:

git fetch origin pull/29609/head:pr-29609                                           
git checkout pr-29609

Thanks!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you need anything else from me here? asking in case if I missed something. As of now, this is not running on M1 Mac.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could try to see what it takes to make it succeed on a Mac but don't spend too much time if you can't get it to work.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good. I will take a look.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i tried few things from stackoverflow and python-snappy github page but no luck so far.

exit 1
fi

if ! dpkg -s libsnappy-dev >/dev/null ; then
echo You must install libsnappy-dev to run this script. Run:
echo sudo apt install libsnappy-dev
fi

if [[ $# -lt 2 ]]; then
printf "Example usage: \n$> ./sdks/python/container/run_generate_requirements.sh 3.8 <sdk_tarball>"
printf "\n\where 3.8 is the Python major.minor version."
Expand Down
Loading