2x import slowdown in latest derivation of Python for macOS with nixpkgs-unstable

[layday]

Describe the bug
After updating nixpkgs (nix-channel --update), there was a noticeable start-up slowdown when invoking several different Python CLIs. Old Python derivation on top, new one on the bottom:

$ /nix/store/xf3244kbn36z04yw98mpgvvsgdcbz438-python3-3.8.8/bin/python -m venv venv && time venv/bin/pip 1&>0
________________________________________________________
Executed in  190.77 millis    fish           external
   usr time  161.01 millis   97.00 micros  160.91 millis
   sys time   25.12 millis  572.00 micros   24.55 millis
$ rm -rf venv
$ /nix/store/rakqkwxql1hsa16dphdfkdky8i9cib5v-python3-3.8.8/bin/python -m venv venv && time venv/bin/pip 1&>0
________________________________________________________
Executed in  509.46 millis    fish           external
   usr time  469.83 millis  100.00 micros  469.73 millis
   sys time   35.16 millis  762.00 micros   34.40 millis

Running python -v revealed that the Python from rakqkwxql1hsa16dphdfkdky8i9cib5v did not have and was unable to create PYC files, meaning that it was unable to compile the standard library. For example:

# code object from /nix/store/rakqkwxql1hsa16dphdfkdky8i9cib5v-python3-3.8.8/lib/python3.8/encodings/__init__.py
# could not create '/nix/store/rakqkwxql1hsa16dphdfkdky8i9cib5v-python3-3.8.8/lib/python3.8/encodings/__pycache__/__init__.cpython-38.pyc': PermissionError(13, 'Permission denied')

Whereas the older Python derivation had:

# /nix/store/xf3244kbn36z04yw98mpgvvsgdcbz438-python3-3.8.8/lib/python3.8/encodings/__pycache__/__init__.cpython-38.pyc matches /nix/store/xf3244kbn36z04yw98mpgvvsgdcbz438-python3-3.8.8/lib/python3.8/encodings/__init__.py
# code object from '/nix/store/xf3244kbn36z04yw98mpgvvsgdcbz438-python3-3.8.8/lib/python3.8/encodings/__pycache__/__init__.cpython-38.pyc'

I expect that this is what's causing the slowdown. Please let me know if there's anything else I can do to help pinpoint the issue.

To Reproduce
As above.

Notify maintainers
@FRidh

Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

• system: "x86_64-darwin"
• host os: Darwin 18.7.0, macOS 10.14.6
• multi-user?: no
• sandbox: no
• version: nix-env (Nix) 2.3.10
• channels(abc): "nixpkgs-21.05pre281282.4cb48cc2562"
• nixpkgs: /Users/abc/.nix-defexpr/channels/nixpkgs

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module:

[FRidh]

That's most likely since 9d03ff5. Unoptimized bytecode is no longer included because it cannot be generated reproducibly (upstream issue https://bugs.python.org/issue34722). Optimized bytecode is available so running with -O or -OO should not give this issue and actually run faster, although some code may rely on code that is removed.

I am of the opinion that reproducibility is more important than performance, but of course the gap here will be huge. Related discussion for gcc #112928. This limitation for sure needs to be documented.

[layday]

You would not want to use -O[O] in development however, and for wrappers (like pip above), you'd have to export PYTHONOPTIMIZE. I think optimisation levels are generally little used in Python. Not sure what to suggest - I've gone in and ran compileall by hand but I'd not want to do that every time there's a new Python derivation nor is it something I'd recommend doing.

[baloo]

Just passing by here, but it looks like @layday is running darwin, and that the optimization were not applied anyway, if I read 9d03ff5#diff-752dca48f3873cc89d448225abb8d9b16d023c01fef41d32f348a410b8ef0a31L41 correctly.

Could it be something else?

[FRidh]

The optimizations build flag not, but I am talking here about bytecode.

[FRidh]

#120858

[andersk]

Unoptimized bytecode is no longer included because it cannot be generated reproducibly (upstream issue https://bugs.python.org/issue34722).

That issue is already negated by our PYTHONHASHSEED=0, no?

[layday]

I think this was fixed a long time ago.