Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Port OpenSSL backend to go1.19 #576

Merged
merged 12 commits into from
May 26, 2022
Merged

Port OpenSSL backend to go1.19 #576

merged 12 commits into from
May 26, 2022

Conversation

qmuntal
Copy link
Member

@qmuntal qmuntal commented May 23, 2022

This PR ports the OpenSSL backend from dev.boringbranch into main following the same generic backend strategy as in previous Go versions.

Upstream Go did several refactors to how BoringCrypto was being integrated into crypto. Our port can leverage almost all work done upstream, except for the following issues:

  • AES and ECDSA keys are cached using a runtime-backed map which does not allocate. The map implementation is located in crypto/internal/boring, and we can't depend on it, so I've copy-pasted the relevant files into crypto/internal/backend/bcache and updated the runtime package to back this new package. See https://groups.google.com/g/golang-dev/c/h6iFPvOnPkg for more context.
  • The whole crypto/internal/boring API is now free of big.Int. It instead crypto/internal/boring.BigInt and a helper package to cast from big.Int to boring.BigInt located at crypto/internal/boring/bbig. As we can't depend on crypto/internal/boring.BigInt, I've created a bidge package at crypto/internal/backend/bbig which implements the casts to the right BigInt type:
  • TestAllocations test from crypto/ed25519 has been disabled when goexperiment.opensslcrypto is set. boring.Enabled is not known at compile time when using OpenSSL backend, as it can be encabled/disabled at runtime, so there are some optimizations that the compiler skip, which make us allocate more even if boring.Enabled==false

@qmuntal qmuntal added the fips label May 23, 2022
@qmuntal qmuntal force-pushed the dev/qmuntal/go1.19 branch from 42cbfd6 to 4a19d3a Compare May 23, 2022 16:25
Copy link
Member

@dagood dagood left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes LGTM, just a couple comments about the patch arrangement.

Comment on lines 251 to 252
--- a/src/crypto/internal/boring/cache.go
+++ /dev/null
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, when I see this, my first thought is that it might belong in the first patch so it's clearer that this is part of a file move. I think Git would also count that as a rename so it would also make the diff clearer (the package name change).

But I remember the idea here is that first patch is add-only (avoiding conflicts), making this deletion part of integration.

Maintenance-wise, I think this might cause a problem down the line. If upstream adds a fix to this file, we'll hit a conflict in this patch, and whoever's resolving it needs to know to not only resolve the conflict in this patch, but also go back to the first patch and redo copying src/crypto/internal/boring/cache.go to src/crypto/internal/backend/bcache/cache.go. It seems like it might be easy to miss and end up not getting the upstream fix in our build.

I think a reasonable middle ground is to split the deletion into a "cleanup" patch file (before or after the integration patch) with a clear description that conflicts need to ported back to the first patch. We don't have a mechanism for it yet, but we could also add a patch test in the outer repo that makes sure the copy is always the same (other than package name).

But actually, I think it might be best to put both the add and remove into the same patch file before even adding the OpenSSL crypto module. Basically, design the patch so that we could even submit it to upstream if they're willing to help us out with https://groups.google.com/g/golang-dev/c/h6iFPvOnPkg. (Maybe it's not quite that simple--the dir structure we want might be different than what upstream would take?) Then, we can simply zap the patch file if upstream merges the change. In the meantime, merge conflicts should be easy to resolve.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the idea of putting this change into its own file. It will also be clearer which refinements we are doing to existing boring package for the sake of reusability.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done! Git does count it as a rename, so we saved almost 500 lines of diffs and future headaches if the original files we changed 😄

Comment on lines 4 to 6
Subject: [PATCH] Vendor OpenSSL crypto library

---
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should the command from the old summary get copied over here?

Subject: [PATCH] Vendor OpenSSL crypto library
To reproduce, run 'go mod vendor' in 'go/src'.
---

More generally about these summaries: I don't think we need to be concerned about including big descriptions for each patch in the initial PR. But, I think we should probably plan to include more in the future, especially as we start to collaborate with other Go distro maintainers. It seems to me like the natural place for people to look to start understanding what we've done.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done! Agree that we should have a small description in each patch, but let's do it in another PR :)

From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
From: Quim Muntal <[email protected]>
Date: Mon, 23 May 2022 12:59:54 +0000
Subject: [PATCH] Adjust Go tests to work with crypto module
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I noticed some changes from https://github.com/microsoft/go/blob/microsoft/dev.boringcrypto.go1.18/patches/0103-Adjust-Go-tests-to-work-with-crypto-module.patch aren't in here. If they're not necessary anymore, that's great, but in case the issues come up in CI, figured I should point it out. 😄

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, I wanted to run the CI without those changes just in case they are not needed anymore, but I'll probably end up adding them 😋

@qmuntal qmuntal force-pushed the dev/qmuntal/go1.19 branch from 4a19d3a to 3ae175a Compare May 24, 2022 13:37
@qmuntal qmuntal force-pushed the dev/qmuntal/go1.19 branch from 3ae175a to 77c268a Compare May 24, 2022 14:02
@qmuntal qmuntal requested review from jaredpar and chsienki May 25, 2022 12:36
@qmuntal qmuntal merged commit 66c1d7c into microsoft/main May 26, 2022
@qmuntal qmuntal deleted the dev/qmuntal/go1.19 branch May 26, 2022 17:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants