Impossible to get runtime with shared library on heroku? #243
Comments
|
ah found builds/README.md |
edmorley
added a commit
that referenced
this issue
May 3, 2022
Shared builds are beneficial for a number of reasons: - Reduces the size of the build, since it avoids the duplication between the Python binary and the static library. For example, for Python 3.10.x on Heroku-20 the, output size drops from 206MB -> 90MB. - Permits use-cases that only work with the shared Python library, and not the static library (such as `pycall.rb` or `PyO3`). - More consistent with the official Python Docker images and other distributions. However, shared builds are slower unless `no-semantic-interposition`and LTO is used: https://fedoraproject.org/wiki/Changes/PythonNoSemanticInterpositionSpeedup It's only as of Python 3.10 that `no-semantic-interposition` is enabled by default, so we only use shared builds on Python 3.10+ to avoid needing to override the default compiler flags. Build size reductions: ``` $ diff -U0 <(cd with-system-expat/ && du -Sh) <(cd shared-lto/ && du -Sh) --- /dev/fd/63 +++ /dev/fd/62 @@ -1 +1 @@ -24M ./bin +56K ./bin @@ -78 +78 @@ -60M ./lib/python3.10/config-3.10-x86_64-linux-gnu +144K ./lib/python3.10/config-3.10-x86_64-linux-gnu @@ -81 +81 @@ -20M ./lib/python3.10/lib-dynload +18M ./lib/python3.10/lib-dynload @@ -103 +103 @@ -59M ./lib +26M ./lib ``` ``` $ diff -U0 <(cd with-system-expat/ && du -sh) <(cd shared-lto/ && du -sh) --- /dev/fd/63 +++ /dev/fd/62 @@ -1 +1 @@ -206M . +90M . ``` Also adds a `du` at the end of the build scripts, to make it easier to see output sizes. Note: This change will only take effect for future Python version releases (or future Heroku stacks) - existing Python binaries are not being recompiled. Configure docs: https://docs.python.org/3/using/configure.html#cmdoption-enable-shared https://docs.python.org/3/using/configure.html#cmdoption-with-lto https://docs.python.org/3/using/configure.html#cmdoption-without-static-libpython Closes #243, #665, #1225. GUS-W-10989125.
edmorley
added a commit
that referenced
this issue
May 4, 2022
Shared builds are beneficial for a number of reasons: - Reduces the size of the build, since it avoids the duplication between the Python binary and the static library. For example, for Python 3.10.x on Heroku-20 the, output size drops from 206MB -> 90MB. - Permits use-cases that only work with the shared Python library, and not the static library (such as `pycall.rb` or `PyO3`). - More consistent with the official Python Docker images and other distributions. However, shared builds are slower unless `no-semantic-interposition`and LTO is used: https://fedoraproject.org/wiki/Changes/PythonNoSemanticInterpositionSpeedup It's only as of Python 3.10 that `no-semantic-interposition` is enabled by default, so we only use shared builds on Python 3.10+ to avoid needing to override the default compiler flags. Build size reductions: ``` $ diff -U0 <(cd with-system-expat/ && du -Sh) <(cd shared-lto/ && du -Sh) --- /dev/fd/63 +++ /dev/fd/62 @@ -1 +1 @@ -24M ./bin +56K ./bin @@ -78 +78 @@ -60M ./lib/python3.10/config-3.10-x86_64-linux-gnu +144K ./lib/python3.10/config-3.10-x86_64-linux-gnu @@ -81 +81 @@ -20M ./lib/python3.10/lib-dynload +18M ./lib/python3.10/lib-dynload @@ -103 +103 @@ -59M ./lib +26M ./lib ``` ``` $ diff -U0 <(cd with-system-expat/ && du -sh) <(cd shared-lto/ && du -sh) --- /dev/fd/63 +++ /dev/fd/62 @@ -1 +1 @@ -206M . +90M . ``` Also adds a `du` at the end of the build scripts, to make it easier to see output sizes. Note: This change will only take effect for future Python version releases (or future Heroku stacks) - existing Python binaries are not being recompiled. Configure docs: https://docs.python.org/3/using/configure.html#cmdoption-enable-shared https://docs.python.org/3/using/configure.html#cmdoption-with-lto https://docs.python.org/3/using/configure.html#cmdoption-without-static-libpython Closes #243, #665, #1225. GUS-W-10989125.
edmorley
added a commit
that referenced
this issue
May 4, 2022
Shared builds are beneficial for a number of reasons: - Reduces the size of the build, since it avoids the duplication between the Python binary and the static library. For example, for Python 3.10.x on Heroku-20 the, output size drops from 206MB -> 90MB. - Permits use-cases that only work with the shared Python library, and not the static library (such as `pycall.rb` or `PyO3`). - More consistent with the official Python Docker images and other distributions. However, shared builds are slower unless `no-semantic-interposition`and LTO is used: https://fedoraproject.org/wiki/Changes/PythonNoSemanticInterpositionSpeedup It's only as of Python 3.10 that `no-semantic-interposition` is enabled by default, so we only use shared builds on Python 3.10+ to avoid needing to override the default compiler flags. Build size reductions: ``` $ diff -U0 <(cd with-system-expat/ && du -Sh) <(cd shared-lto/ && du -Sh) --- /dev/fd/63 +++ /dev/fd/62 @@ -1 +1 @@ -24M ./bin +56K ./bin @@ -78 +78 @@ -60M ./lib/python3.10/config-3.10-x86_64-linux-gnu +144K ./lib/python3.10/config-3.10-x86_64-linux-gnu @@ -81 +81 @@ -20M ./lib/python3.10/lib-dynload +18M ./lib/python3.10/lib-dynload @@ -103 +103 @@ -59M ./lib +26M ./lib ``` ``` $ diff -U0 <(cd with-system-expat/ && du -sh) <(cd shared-lto/ && du -sh) --- /dev/fd/63 +++ /dev/fd/62 @@ -1 +1 @@ -206M . +90M . ``` Also adds a `du` at the end of the build scripts, to make it easier to see output sizes. Note: This change will only take effect for future Python version releases (or future Heroku stacks) - existing Python binaries are not being recompiled. Configure docs: https://docs.python.org/3/using/configure.html#cmdoption-enable-shared https://docs.python.org/3/using/configure.html#cmdoption-with-lto https://docs.python.org/3/using/configure.html#cmdoption-without-static-libpython Closes #243, #665, #1225. GUS-W-10989125.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I need to get a version of Python working with shared library support. I've tried a number of things:
heroku-buildpack-pythonand specifiedpython-2.7.8-sharedinruntime.txt. This gives me 2.7.8 allright. But in/app/.heroku/python/libI still findlibpython2.7.a, so it's useless.python-2.7.7-shared. Gives me an error message"Requested runtime (python-2.7.7-shared) is not available for this stack (cedar-14)."python-2.7.10-sharedwith the--enable-sharedcompilation flag. Then I told Heroku to use my own forked buildpack. Again when I deploy, it gives me an error message"Requested runtime (python-2.7.7-shared) is not available for this stack (cedar-14)."I'm also not sure whether Heroku actually downloads and compiles a new Python runtime, or takes a runtime from some pre compiled cache?
The text was updated successfully, but these errors were encountered: