Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

protobuf: 24.4 -> 25.3 #264902

Merged
merged 1 commit into from
May 22, 2024
Merged

protobuf: 24.4 -> 25.3 #264902

merged 1 commit into from
May 22, 2024

Conversation

GaetanLepage
Copy link
Contributor

@GaetanLepage GaetanLepage commented Nov 1, 2023

Description of changes

  • Add protobuf v25.2.
  • Set it as the default version (previously v24.4).

Note: I had to manually set the abseil version to a more recent one (202308) instead of the default one (202301).
Maybe it should be consider to bump the default abseil-cpp to a newer version.

Changelog: https://github.com/protocolbuffers/protobuf/releases/tag/v25.2

cc @tobim @jonringer

Things done

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 23.11 Release Notes (or backporting 23.05 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

@tobim
Copy link
Contributor

tobim commented Nov 2, 2023

I think we should wait until 23.11 is out before doing this update.

Note: I had to manually set the abseil version to a more recent one (202308) instead of the default one (202301).
Maybe it should be consider to bump the default abseil-cpp to a newer version.

Indeed, this pin would result in many packages to have multiple version of abseil in their closures. On paper abseil supports that, but even if it doesn't cause any build failures it would still be a waste of resources, imho.

@GaetanLepage
Copy link
Contributor Author

GaetanLepage commented Nov 2, 2023

I think we should wait until 23.11 is out before doing this update.

Fine by me !

Indeed, this pin would result in many packages to have multiple version of abseil in their closures. On paper abseil supports that, but even if it doesn't cause any build failures it would still be a waste of resources, imho.

The right thing to do would be to bump the default version of abseil to a more recent one right ?

@GaetanLepage GaetanLepage changed the title protobuf: 24.4 -> 25.0 protobuf: 24.4 -> 25.1 Nov 15, 2023
@GaetanLepage
Copy link
Contributor Author

@tobim in the end I bumped the default abseil version.

@GaetanLepage
Copy link
Contributor Author

python311Packages.protobuf fails to build with

==================================== ERRORS ====================================
_______________________ ERROR collecting minimal_test.py _______________________
ImportError while importing test module '/build/source/python/minimal_test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/nix/store/480sm7i0dn11hf0lmx4wl4gk9w1k5s8r-python3-3.11.6/lib/python3.11/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
minimal_test.py:35: in <module>
    from google.protobuf.pyext import _message
E   ImportError: cannot import name '_message' from 'google.protobuf.pyext' (/build/source/python/google/protobuf/pyext/__init__.py)

@GaetanLepage
Copy link
Contributor Author

python311Packages.protobuf fails to build with

protocolbuffers/protobuf#14880 (comment)

@wegank wegank mentioned this pull request Dec 6, 2023
13 tasks
@de11n
Copy link

de11n commented Dec 6, 2023

It would be more convenient if we could add protobuf3_25 without bumping the default so that folks don't have to wait to use it.

@GaetanLepage GaetanLepage marked this pull request as draft December 6, 2023 15:40
@de11n
Copy link

de11n commented Dec 6, 2023

It would be more convenient if we could add protobuf3_25 without bumping the default so that folks don't have to wait to use it.

Thank you! C.f. #272493

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So the README says that

  • the cpp (.../pyext) backend is "legacy" and advises to use the μpb backend,
  • mentions that

    While upb offers a C API, the C API & ABI are not stable. For this reason, upb is not generally offered as a C library for direct consumption, and there are no releases. https://github.com/protocolbuffers/protobuf/tree/main/upb

Did we ever have any ABI compatibility guarantees with cpp? I'm wondering if we could do any tricks for tensorflow like build it with the older protobuf but link the newer one, and also maybe use nixpkgs-pytools to rename its references to the old protobuf python module? (only relevant until we update tf to 2.15?)

@ofborg ofborg bot requested a review from knedlsepp February 19, 2024 17:09
knedlsepp added a commit to knedlsepp/nixpkgs that referenced this pull request May 19, 2024
In order to enable bumping the default protobuf version from protobuf_24
to protobuf_25, we address the build failure of pythonPackages.protobuf
against that version.
Protobuf's python package is moving away from cpp backend in favor of a
µpb backend.
(See: https://github.com/protocolbuffers/protobuf/tree/main/upb)
The work on that seems to have lead to the introduction of a broken test
in "minimal_test.py":
protocolbuffers/protobuf@501ecec
I suspect that this is not an issue on the nixpkgs packaging end but
rather that this file is uncovered code upstream. I don't know enough
about Bazel to be sure, but it looks like that file is not part of their
protobuf/python/BUILD.bazel file.
(I wanted to prove that in
protocolbuffers/protobuf#16888 but couldn't
trigger upstream's CI)
So for now let's just skip that file.
Note that protobuf_26.tests.pythonProtobuf is still broken. This is due
to the fact that upstream removed support for building the library
directly from the GitHub repo. (See:
protocolbuffers/protobuf#15708)
Conda packaging is also currently struggling with this:
conda-forge/protobuf-feedstock#215

Related: NixOS#264902
@GaetanLepage GaetanLepage requested a review from SomeoneSerge May 20, 2024 15:24
@GaetanLepage GaetanLepage changed the title protobuf: 24.4 -> 25.2 protobuf: 24.4 -> 25.3 May 20, 2024
@ofborg ofborg bot requested a review from happysalada May 20, 2024 17:16
@GaetanLepage
Copy link
Contributor Author

@ofborg build protobuf.tests

Copy link
Contributor

@SomeoneSerge SomeoneSerge left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's safe to merge now. This won't go into the release but into the unstable

@wegank wegank added the 12.approvals: 2 This PR was reviewed and approved by two reputable people label May 22, 2024
@SomeoneSerge
Copy link
Contributor

RE: ofborg

Hypothesis doesn't depend on protobuf and so is broken independently:

❯ nix path-info --recursive --derivation nixpkgs#python311Packages.hypothesis | grep protobuf
~

@SomeoneSerge SomeoneSerge merged commit c24592c into NixOS:staging May 22, 2024
31 checks passed
@GaetanLepage GaetanLepage deleted the protobuf branch May 22, 2024 12:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants