curl: 7.76.1 -> 7.77.0

[mweinelt]

Motivation for this change

https://curl.se/changes.html\#7_77_0

Fixes: CVE-2021-22897, CVE-2021-22898, CVE-2021-22901

Things done

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Added a release notes entry if the change is major or breaking
• Fits CONTRIBUTING.md.

[mweinelt]

We should probably backport this one to 20.09:

• https://curl.se/docs/CVE-2021-22901.html
  curl/curl@7f4a9a9

These two I feel are optional:

• https://curl.se/docs/CVE-2021-22897.html
  curl/curl@bbb7150
• https://curl.se/docs/CVE-2021-22898.html
  curl/curl@39ce47f

[lukegb]

Testing a build of curl, nix, and running a smattering of NixOS tests to boot. Will report back.

[lukegb]

curl, nix, and the simple test built. The installer tests also seem fine, so

[SuperSandro2000]

I think this update might break darwin:

checking for gethostbyname in -lsocket... no
checking for gethostbyname in -lwatt... no
checking for gethostbyname with both nsl and socket libs... no
checking for gethostbyname for Minix 3... no
checking for gethostbyname for eCos... no
checking for gethostbyname for AmigaOS bsdsocket.library... no
checking for gethostbyname in -lnetwork... no
checking for gethostbyname in -lnet... no
configure: error: couldn't find libraries for gethostbyname()
builder for '/nix/store/jvamzd2h2k0rpqagplyjs68645v9wj7x-curl-7.77.0.drv' failed with exit code 1

https://logs.nix.ci/?key=nixos/nixpkgs.124502&attempt_id=98158734-bf8f-4dbd-aa2a-abc9a125a501

@ofborg build curl

/cc @NixOS/darwin-maintainers

[risicle]

Can confirm @SuperSandro2000 's failure, macos 10.15, sandbox enabled.

[risicle]

The diff between the two curls https://github.com/curl/curl/compare/curl-7_76_1..curl-7_77_0 suggests that --with-openssl is now preferred over --with-ssl. Similarly --without-....

[mweinelt]

Migrated to --with-openssl and from sslSupport to opensslSupport treewide. But that probably doesn't fix anything on darwin, no?

[risicle]

No it doesn't, still 👀 and 🤔

[risicle]

Yup, digging into the config.log it now wants the SystemConfiguration framework for darwin:

configure:20943: checking for gethostbyname
configure:20943: clang -o conftest -Qunused-arguments -Os -Werror=partial-availability   -framework SystemConfiguration conftest.c  >&5
ld: framework not found SystemConfiguration

the problem with this being that adding it causes a reference loop.

[risicle]

SystemConfiguration dependency added for NAT64 support in curl/curl#7121

[risicle]

Created #124982 in case we want to take that route.

[mweinelt]

I feel like we do, thanks for investing the time. Did anyone mention the issue in #macos:nixos.org yet?

[risicle]

Yes, the conclusion was that this is indeed the first time a macos framework in included in the curl loop, so there's no prior art.

I think there are two possibilities:

a) hack the curl build system to allow building a SystemConfiguration-less version. Easiest solution short-term.. unpredictable maintenance burden long-term depending on curl upstream changes?
b) create an overridden, bootstrappable SystemConfiguration. Having a little look at this at the moment.

[risicle]

Option b) is tricky because apple sdk needed for framework's headers > comes in xar format > has xml-based metadata > libxml2 > python > expat ...

[andir]

cc @NixOS/darwin-maintainers

[thefloweringash]

b) create an overridden, bootstrappable SystemConfiguration. Having a little look at this at the moment.

This is my preferred solution. The more our build environment diverges from the expected build environment, the more likely problems like this will arise.

If we include pbzx in the bootstrap tools, we can start building with anything we require from the sdk immediately.

A start in that direction

diff --git a/pkgs/os-specific/darwin/apple-sdk/default.nix b/pkgs/os-specific/darwin/apple-sdk/default.nix
index 1b60abf562b..3cc9bd8e899 100644
--- a/pkgs/os-specific/darwin/apple-sdk/default.nix
+++ b/pkgs/os-specific/darwin/apple-sdk/default.nix
@@ -16,27 +16,20 @@ let
       sha256 = "13xq34sb7383b37hwy076gnhf96prpk1b4087p87xnwswxbrisih";
     };
 
-    nativeBuildInputs = [ xar cpio python3 pbzx ];
+    nativeBuildInputs = [ cpio pbzx ];
 
     outputs = [ "out" "dev" "man" ];
 
     unpackPhase = ''
-      xar -x -f $src
+      pbzx $src | cpio -idm
     '';
 
     installPhase = ''
-      start="$(pwd)"
-      mkdir -p $out
-      cd $out
-      pbzx -n $start/Payload | cpio -idm
+      mkdir $out
 
-      mv usr/* .
-      rmdir usr
+      cp -r System/* usr/* $out
 
-      mv System/* .
-      rmdir System
-
-      pushd lib
+      pushd $out/lib
       cp ${darwin-stubs}/usr/lib/libcups*.tbd .
       ln -s libcups.2.tbd      libcups.tbd
       ln -s libcupscgi.1.tbd   libcupscgi.tbd
diff --git a/pkgs/stdenv/darwin/make-bootstrap-tools.nix b/pkgs/stdenv/darwin/make-bootstrap-tools.nix
index 3af444a2e52..cd7c0fb9980 100644
--- a/pkgs/stdenv/darwin/make-bootstrap-tools.nix
+++ b/pkgs/stdenv/darwin/make-bootstrap-tools.nix
@@ -97,16 +97,16 @@ in rec {
       mkdir $out/include
       cp -rd ${llvmPackages.libcxx.dev}/include/c++     $out/include
 
+      # copy package extraction tools
+      cp -d ${pkgs.pbzx}/bin/pbzx $out/bin
+      cp -d ${pkgs.xar}/lib/libxar*.dylib $out/lib
+      cp -d ${pkgs.bzip2.out}/lib/libbz2*.dylib $out/lib
+
       ${lib.optionalString targetPlatform.isAarch64 ''
         # copy .tbd assembly utils
         cp -d ${pkgs.darwin.rewrite-tbd}/bin/rewrite-tbd $out/bin
         cp -d ${pkgs.libyaml}/lib/libyaml*.dylib $out/lib
 
-        # copy package extraction tools
-        cp -d ${pkgs.pbzx}/bin/pbzx $out/bin
-        cp -d ${pkgs.xar}/lib/libxar*.dylib $out/lib
-        cp -d ${pkgs.bzip2.out}/lib/libbz2*.dylib $out/lib
-
         # copy sigtool
         cp -d ${pkgs.darwin.sigtool}/bin/sigtool $out/bin
         cp -d ${pkgs.darwin.sigtool}/bin/codesign $out/bin

[LnL7]

If possible I would rather avoid adding more things to bootstrap tools since anything in there is rather annoying to update.

Since curl itself is part of bootstrap tools already however. If I'm not mistaken we only need curl as native input and don't link against it, if that's the case the bootstrap version can be propagated much further in the stdenv stages possibly eliminating curl as a dependency alltogether.

[vcunat]

I don't know... this feels like curl on all platform would get blocked due to this issue. What about... NAT64 not working on darwin in curl? (like so far, presumably) That commit isn't so complex, so e.g. simply revert it for darwin? (maybe just during bootstrapping) I mean, until someone implements some better solution like discussed above, but who knows how long that might take.

[SuperSandro2000]

Something needs to be done about this update.
Some options I see right now (partly copying what vcunat already wrote above):

• drop the commit that is causing issues on darwin
• use an older curl version for darwin (bootstraping)

Blocking this important update for more than 3 months just because Darwin stdenv is way behind and not easy to maintain is a pity but it shouldn't hold back Linux and create a lot of extra work for all people maintaining the Linux part of nixpkgs and NixOS.

[r-burns]

Has anyone mentioned this to upstream? I'm sure a project as low-level as curl would be sympathetic to this breakage and be willing to provide a blessed path for systemconfiguration-less building.

[risicle]

My guess is that most macos developers wouldn't see the benefit of trying to build without SystemConfiguration.

[SuperSandro2000]

Has anyone mentioned this to upstream? I'm sure a project as low-level as curl would be sympathetic to this breakage and be willing to provide a blessed path for systemconfiguration-less building.

Most apple users are not using a very old SDK to build curl.

[lheckemann]

Superseded by #141829. Feel free to reopen if I've misunderstood.