Firefox shows public certificate authorities as “not recognized by Mozilla”

[sigprof]

Describe the bug
Firefox (from pkgs.firefox) shows “Connection verified by a certificate issuer that is not recognized by Mozilla” even for well-known public certificate authorities (DigiCert, Let's Encrypt, …).

To Reproduce
Steps to reproduce the behavior:

1. Install Firefox on NixOS 21.05 from the unmodified pkgs.firefox package.
2. Start Firefox with an empty profile; it helpfully opens the “Firefox Privacy Notice” page from https://www.mozilla.org/en-US/privacy/firefox/ (you can also open https://addons.mozilla.org, https://www.nixos.org/, …)
3. Click the padlock icon to the left of the URL bar to open the site information popup.
4. Observe the warning: “Connection verified by a certificate issuer that is not recognized by Mozilla”.

Expected behavior
Firefox should not show warnings about not recognized certificate issuers when the site certificate was issued by a well-known public certificate authority (not by some local CA added via policy).

Screenshots

Additional context
Tested Firefox 89.0 at the most recent nixos-21.05 commit at the moment (5de44c1), 88.0.1 @ d6229c7, 87.0 @ 29058f9 — they are all affected; not sure whether it is possible to test some older versions without the need to build them from source. Firefox 78.11.0esr (pkgs.firefox-esr) does not have the problem, but I'm not sure whether that older version is actually able to show that warning.

Apparently this is caused by the nss change to use p11-kit (#96763). Just for testing I tried to build nss without that option:

nix profile install --impure --expr '(import <nixpkgs> {}).nss.override { useP11kit = false; }'

When I inject that library version into Firefox using LD_PRELOAD=$HOME/.nix-profile/lib/libnss3.so, the warning about “certificate issuer that is not recognized by Mozilla” disappears.

Note that the problems in Firefox are apparently not limited to that warning — extension installation using extraPolicies.ExtensionSettings with install_url pointing to https://addons.mozilla.org/ also does not work, showing messages complaining about unrecognized certificates. The addon manager, however, still works even in this configuration.

Notify maintainers
@rnhmjoj
@ajs124
@mweinelt
@lovesegfault

Metadata

• system: "x86_64-linux"
• host os: Linux 5.10.37, NixOS, 21.05.20210604.aa57635 (Okapi)
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.4pre20210601_5985b8b
• nixpkgs: /nix/store/zjmnwn76xzxw3j7v1p98fd0aywpvyylf-source

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module:

[mweinelt]

Right, so nss uses p11kit since #96763 to make every nss consumer rely on the system certificate store. We could rebuild firefox with the nss override you mentioned, that would then decouple firefox from the central certificate store, but the kicker is, that our cacert package is extracted from nss itself, so it is in fact the same IIUC. So it might make sense to look into how they certify those, that makes Firefox see them as recognized.

[mweinelt]

  /**
   * Returns whether the issuer of the current certificate chain is
   * built-in (returns false) or imported (returns true).
   */
  _hasCustomRoot() {
    let issuerCert = null;
    issuerCert = this._secInfo.succeededCertChain[
      this._secInfo.succeededCertChain.length - 1
    ];

    return !issuerCert.isBuiltInRoot;
  },

browser/base/content/browser-siteIdentity.js

Looks to be this isBuiltInRoot attribute, which is ultimately hooked into this function:

// The term "builtin root" traditionally refers to a root CA certificate that
// has been added to the NSS trust store, because it has been approved
// for inclusion according to the Mozilla CA policy, and might be accepted
// by Mozilla applications as an issuer for certificates seen on the public web.
Result IsCertBuiltInRoot(CERTCertificate* cert, bool& result) {
  if (NS_FAILED(BlockUntilLoadableCertsLoaded())) {
    return Result::FATAL_ERROR_LIBRARY_FAILURE;
  }

  result = false;
#ifdef DEBUG
  nsCOMPtr<nsINSSComponent> component(do_GetService(PSM_COMPONENT_CONTRACTID));
  if (!component) {
    return Result::FATAL_ERROR_LIBRARY_FAILURE;
  }
  nsresult rv = component->IsCertTestBuiltInRoot(cert, &result);
  if (NS_FAILED(rv)) {
    return Result::FATAL_ERROR_LIBRARY_FAILURE;
  }
  if (result) {
    return Success;
  }
#endif  // DEBUG
  AutoSECMODListReadLock lock;
  for (SECMODModuleList* list = SECMOD_GetDefaultModuleList(); list;
       list = list->next) {
    for (int i = 0; i < list->module->slotCount; i++) {
      PK11SlotInfo* slot = list->module->slots[i];
      // We're searching for the "builtin root module", which is a module that
      // contains an object with a CKA_CLASS of CKO_NETSCAPE_BUILTIN_ROOT_LIST.
      // We use PK11_HasRootCerts() to identify a module with that property.
      // In the past, we exclusively used the PKCS#11 module named nssckbi,
      // which is provided by the NSS library.
      // Nowadays, some distributions use a replacement module, which contains
      // the builtin roots, but which also contains additional CA certificates,
      // such as CAs trusted in a local deployment.
      // We want to be able to distinguish between these two categories,
      // because a CA, which may issue certificates for the public web,
      // is expected to comply with additional requirements.
      // If the certificate has attribute CKA_NSS_MOZILLA_CA_POLICY set to true,
      // then we treat it as a "builtin root".
      if (PK11_IsPresent(slot) && PK11_HasRootCerts(slot)) {
        CK_OBJECT_HANDLE handle = PK11_FindCertInSlot(slot, cert, nullptr);
        if (handle != CK_INVALID_HANDLE &&
            PK11_HasAttributeSet(slot, handle, CKA_NSS_MOZILLA_CA_POLICY,
                                 false)) {
          // Attribute was found, and is set to true
          result = true;
          break;
        }
      }
    }
  }
  return Success;
}

security/certverifier/CertVerifier.cpp

So it comes down to the CKA_NSS_MOZILLA_CA_POLICY attribute on a certificate. Why do we loose that when using p11-kit? I guess this could be because of certdata2pem.py used in the cacert package.

But apparently p11-kit should support CKA_NSS_MOZILLA_CA_POLICY, see https://github.com/p11-glue/p11-kit/pull/46/files. So maybe it is a problem in our p11-kit packaging.

[ajs124]

The reason why this doesn't happen with firefox esr 78 is that that uses an older version of nss, in which we didn't have p11-kit support yet.

[sigprof]

So maybe it is a problem in our p11-kit packaging.

Looks like it. In nixpkgs p11-kit uses --with-trust-paths=/etc/ssl/certs/ca-certificates.crt — this is apparently a plain OpenSSL-compatible certificate bundle which does not contain any certificate attributes. But in Fedora p11-kit is configured like this (they build it with meson):

# These paths are the source paths that  come from the plan here:
# https://fedoraproject.org/wiki/Features/SharedSystemCertificates:SubTasks
%meson -Dgtk_doc=true -Dman=true -Dtrust_paths=%{_sysconfdir}/pki/ca-trust/source:%{_datadir}/pki/ca-trust-source

/usr/share/pki/ca-trust-source comes from the ca-certificates package, and contains ca-bundle.trust.p11-kit, which looks like this:

# This is a bundle of X.509 certificates of public Certificate
# Authorities.  It was generated from the Mozilla root CA list.
# These certificates and trust/distrust attributes use the file format accepted
# by the p11-kit-trust module.
#
# Source: nss/lib/ckfw/builtins/certdata.txt
# Source: nss/lib/ckfw/builtins/nssckbi.h
#
# Generated from:
# NSS_BUILTINS_LIBRARY_VERSION "2.48"
#
[p11-kit-object-v1]
label: "ACCVRAIZ1"
class: x-certificate-extension
object-id: 2.5.29.37
value: "0 %06%03U%1d%25%01%01%ff%04%160%14%06%08%2b%06%01%05%05%07%03%04%06%08%2b%06%01%05%05%07%03%01"
modifiable: false
-----BEGIN PUBLIC KEY-----
MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAm6mrv2FKl68vl2aadF/Q
...
-----END PUBLIC KEY-----

[p11-kit-object-v1]
label: "ACCVRAIZ1"
trusted: true
nss-mozilla-ca-policy: true
modifiable: false
nss-server-distrust-after: "%00"
nss-email-distrust-after: "%00"
-----BEGIN CERTIFICATE-----
MIIH0zCCBbugAwIBAgIIXsO3pkN/pOAwDQYJKoZIhvcNAQEFBQAwQjESMBAGA1UE
...

Note nss-mozilla-ca-policy: true — this format supports such certificate attributes. Fedora also has some scripts in their packages that can extract the certificate information from that format and write it out in various other formats, such as OpenSSL-compatible certificate bundle, Java key store, …

[ajs124]

(they build it with meson):

There was an attempt to build p11-kit with meson before, in #73341, although that seems to have stalled.

/usr/share/pki/ca-trust-source comes from the ca-certificates package, and contains ca-bundle.trust.p11-kit

So maybe we should also modify our cacert package to include this?

On a side-note, Arch Linux also seems to use a different cacertdata2pem.py script from us, which looks like it generates a file like this.

[sigprof]

On a side-note, Arch Linux also seems to use a different cacertdata2pem.py script from us, which looks like it generates a file like this.

Yes, and apparently it's even the same script as the one used in Fedora.

So maybe we should also modify our cacert package to include this?

Yes, this would make sense. Fedora treats the p11-kit format as the master one, and generates all other formats from it:

/usr/bin/p11-kit extract --format=openssl-bundle --filter=certificates --overwrite --comment $DEST/openssl/ca-bundle.trust.crt
/usr/bin/p11-kit extract --format=pem-bundle --filter=ca-anchors --overwrite --comment --purpose server-auth $DEST/pem/tls-ca-bundle.pem
/usr/bin/p11-kit extract --format=pem-bundle --filter=ca-anchors --overwrite --comment --purpose email $DEST/pem/email-ca-bundle.pem
/usr/bin/p11-kit extract --format=pem-bundle --filter=ca-anchors --overwrite --comment --purpose code-signing $DEST/pem/objsign-ca-bundle.pem
/usr/bin/p11-kit extract --format=java-cacerts --filter=ca-anchors --overwrite --purpose server-auth $DEST/java/cacerts
/usr/bin/p11-kit extract --format=edk2-cacerts --filter=ca-anchors --overwrite --purpose=server-auth $DEST/edk2/cacerts.bin

Doing it like this, however, would bring p11-kit into the dependency chain of cacerts, which might not be desired.

And we will need to do something with nixos/modules/security/ca.nix — these options should also affect the contents of ca-bundle.trust.p11-kit somehow.

[mweinelt]

Agreed, let's try that route (the fedora one), we are also leaning on them for firefox sometimes and they have dedicated people for this stuff.

[ajs124]

I'm not sure if and when I'll have some time to further look into this. Do any of you think you'll get around to working on this in the foreseeable future?

Also, we should probably get the cacert maintainers in here cc @andir @lukegb

[lukegb]

I'll try to look at this this evening.

[dasJ]

I don't see this mentioned here but it's currently not possible to install add-ons or to update them. This also includes the automatic update functionality.

Installing an add-on from AMO:

Running the auto-updater from about:addons top-right cogwheel tells me there are no updates.
However opening the browser console from the developer menu shows something like:

1623187372132 addons.update-checker WARN Request failed: https://versioncheck.addons.mozilla.org/update/VersionCheck.php?reqVersion=2&[email protected]&version=0.6.3&maxAppVersion=*&status=userDisabled&appID={ec8030f7-c20a-464f-9b0e-13a3a9e97384}&appVersion=88.0.1&appOS=Linux&appABI=x86_64-gcc3&locale=en-US&currentAppVersion=88.0.1&updateType=97&compatMode=normal - [Exception... "Certificate issuer is not built-in." nsresult: "0x80004004 (NS_ERROR_ABORT)" location: "JS frame :: resource://gre/modules/CertUtils.jsm :: checkCert :: line 183" data: no]

This is a critical security problem since plugins with vulnerabilities are not updateable without manual work to disable certificate verification. cc @NixOS/security-notifications

@ajs124 committed a downstream fix workaround for this in our overlay (hope it's okay to share ;):

self: super: {
  firefox = super.firefox.override {
    nss = unstable.nss.override { useP11kit = false; };
  };
}

[sigprof]

So at least the emergency fix is in nixos-21.05 now, and seems to work — there is no scary warning about unrecognized certificate issuer, and addon installation and updating seems to work (I did not immediately notice this part of the problem, because I was using an existing profile with the needed addons already installed). Did not test the extraPolicies.ExtensionSettings part yet, but I guess that the addon installation would work too.

Should I just close this issue now (because the Firefox part has been handled), or would you prefer to keep it open, even though the problem in the form described at the first message does not exist anymore?

Also I just found #8247, where this change should probably be noted too.

[rnhmjoj]

The issue with Firefox is solved by simply turning off p11-kit integration, it 100% undoes the change in #96763. So, yeah le'ts close this.

[mweinelt]

AFAIU andir and lukgeb are working on updating the cacert/p11-kit packaging so that we can eventually reenable the integration.

[felixsinger]

Why was the issue closed? In my opinion, disabling p11-kit is not a good long-term solution. What is the status of the cacert / p11-kit packaging? Any updates?

[mweinelt]

It isn't the chosen long-term solution, but the issue itself is resolved.

The long-term solution will be #136777.

[felixsinger]

Thanks!