Validatable release artifacts are not sufficient for packaging (trying to run tests)

[dvzrv]

Description

Hi! 👋
I package this project for Arch Linux.

The current signed source artifacts attached to releases are unfortunately not enough to allow me to run tests.
I would like to be able to run tests on source artifacts, that signed, for this I am currently relying on the signed tags, but 3.11.3 has not been a signed tag, so I have been unable to upgrade to that version yet.

What I am currently doing is to checkout a signed tag, verify it and then stage in the test data for running the tests.
https://gitlab.archlinux.org/archlinux/packaging/packages/nlohmann-json/-/blob/314ac9199c8f76d5b661dc0fd79caf3f4f8e1cb8/PKGBUILD

The signed artifacts unfortunately miss a bunch of the files that I would need for packaging (e.g. license files, tests, etc.).

Would it be possible to either use signed tags again or to create a source artifact that includes just the git repository's contents?

Reproduction steps

Try to run tests from one of the signed artifacts in https://github.com/nlohmann/json/releases/tag/v3.11.3

Expected vs. actual results

I am able to run tests from a validatable (signed) source artifact.

Currently I am not able to run tests from a validatable (signed) source artifact.

Minimal code example

No response

Error messages

No response

Compiler and operating system

Arch Linux, gcc 13.2.1

Library version

3.11.3

Validation

• The bug also occurs if the latest version from the develop branch is used.
• I can successfully compile and run the unit tests.

[dvzrv]

@nlohmann would you have some input for me here? I am currently blocked from upgrading the package on Arch Linux due to this issue

[nlohmann]

I am rather unfamiliar what I can do to help here.

[dvzrv]

I am rather unfamiliar what I can do to help here.

To quote myself from above 😄

Would it be possible to either use signed tags again or to create a source artifact that includes just the git repository's contents?

[nlohmann]

But isn't https://github.com/nlohmann/json/releases/tag/v3.11.3 signed?

[dvzrv]

Yes, the tag commit is signed, but by the OpenPGP key that anyone with a github account has access to. It is unfortunately useless for the purpose of authenticating a person (you) and any claims they make (e.g. a signature):

gpg --list-key 4AEE18F83AFDEB23
pub   rsa2048/4AEE18F83AFDEB23 2017-08-16 [SC] [expired: 2024-01-16]
      5DE3E0509C47EA3CF04A42D34AEE18F83AFDEB23
uid                 [ expired] GitHub (web-flow commit signing) <[email protected]>

The artifacts attached to that release are signed with your personal OpenPGP key:

gpg --verify include.zip.asc
gpg: assuming signed data in 'include.zip'
gpg: Signature made 2023-11-28T22:00:26 CET
gpg:                using RSA key 797167AE41C0A6D9232E48457F3CEA63AE251B69
gpg: Good signature from "Niels Lohmann <[email protected]>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 7971 67AE 41C0 A6D9 232E  4845 7F3C EA63 AE25 1B69

We are locking the fingerprint of your personal OpenPGP certificate and use that exact certificate to verify artifacts, commits or tags in the context of building a package for Arch Linux:

https://gitlab.archlinux.org/archlinux/packaging/packages/nlohmann-json/-/blob/314ac9199c8f76d5b661dc0fd79caf3f4f8e1cb8/PKGBUILD#L24

Prior to 3.11.3 this worked and I was able to build and test from a signed tag commit (that I can verify):
https://gitlab.archlinux.org/archlinux/packaging/packages/nlohmann-json/-/blob/314ac9199c8f76d5b661dc0fd79caf3f4f8e1cb8/PKGBUILD#L17

The 3.11.3 release I can not verify, because it is not signed with your personal OpenPGP key and the (signed) artifacts attached to that release do not allow me to run tests (because there are none in them).

Hence this ticket.

[dvzrv]

ERRATA:

The v3.11.3 tag is in fact not signed. The commit it points at is though (by the github OpenPGP key):

$ git ls-remote -t https://github.com/nlohmann/json |rg v3.11.3
9cca280a4d0ccf0c08f47a99aa71d1b0e52f8d03        refs/tags/v3.11.3

$ git verify-tag v3.11.3
error: v3.11.3: cannot verify a non-tag object of type commit.

$ git verify-commit 9cca280a4d0ccf0c08f47a99aa71d1b0e52f8d03
gpg: Signature made 2023-11-28T22:36:31 CET
gpg:                using RSA key 4AEE18F83AFDEB23
gpg: Good signature from "GitHub (web-flow commit signing) <[email protected]>" [expired]
gpg: Note: This key has expired!
Primary key fingerprint: 5DE3 E050 9C47 EA3C F04A  42D3 4AEE 18F8 3AFD EB23

Comparing this to the previous release, it appears you are using annotated tags (git tag -a), and not signed tags (git tag -s) - see https://man.archlinux.org/man/git-tag.1#OPTIONS for an overview.

$ git ls-remote -t https://github.com/nlohmann/json |rg v3.11.2
0ca0fe433eb70cea0d5761079c0c5b47b736565b        refs/tags/v3.11.2
bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d        refs/tags/v3.11.2^{}

$ git verify-tag v3.11.2
error: no signature found

$ git verify-commit bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d
gpg: Signature made 2022-08-12T15:06:06 CEST
gpg:                using RSA key 797167AE41C0A6D9232E48457F3CEA63AE251B69
gpg: Good signature from "Niels Lohmann <[email protected]>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 7971 67AE 41C0 A6D9 232E  4845 7F3C EA63 AE25 1B69

It would be great if signed tags would be created using your personal OpenPGP key and / or signed artifacts containing a snapshot of the entire repository could be attached to releases.
Both would help me to a) verify your signature and b) run tests.

[dvzrv]

After spending some time writing this I'd kindly ask you to please use signed tags, as it is quite likely that we will prefer plain git sources over custom source tarballs in the future.

I don't know what your current release workflow is, but this would basically be down to using git tag -s <tag> instead of git tag or git tag -a.

[dvzrv]

ping @nlohmann

[dvzrv]

@nlohmann can you please respond to this? It is blocking me from upgrading the package on Arch Linux 😰

[tpkessler]

@nlohmann Do you have an update for us on this issue? As @dvzrv said, we'd like to avoid using unsigned tar archives. With git, you can use David's suggestion:

I don't know what your current release workflow is, but this would basically be down to using git tag -s <tag> instead of git tag or git tag -a.

[tpkessler]

Friendly ping @nlohmann :) As discussed above, you'd do us and the Arch community a great favor if you'd properly sign your releases.

[nlohmann]

I currently plan no release. I will get back to you before next time to understand what you need.

[dvzrv]

@nlohmann: I think it would be best for you to not sign release artifacts at this point to be honest.
It is clear that this is not of priority to this project and that the reasons as to why OpenPGP signing is used for the various artifacts is unclear (or rather that there is no real commitment to doing so).

As this has been blocking our package upgrade for nearly half a year now, I will drop the OpenPGP verification from the package and also not add it back unless there is a strong commitment from your side.
That said: If you have questions about OpenPGP signing useful to downstream consumers (also for testing), I'd be happy to help. Feel free to reach out to me (contact is in my profile).

Other than that, I consider this issue closed, so that I can finally move on and upgrade the package. 👋