ACME test is nondeterministic

[emilazy]

This flakiness is "well-known" by now, see e.g. #85185 (comment), #85223 (comment); there's systemd unit dependency issues in the test.

@yegortimoshenko has a patch that improves this, not sure if it fixes it entirely.

[lukateras]

Here's the diff:

diff --git a/nixos/tests/acme.nix b/nixos/tests/acme.nix
index 638601cd75f..e2bcb419f14 100644
--- a/nixos/tests/acme.nix
+++ b/nixos/tests/acme.nix
@@ -51,10 +51,13 @@ in import ./make-test-python.nix {
             webroot = "/var/lib/acme/acme-challenges";
         };
       };
-      systemd.targets."acme-finished-standalone.com" = {};
+      systemd.targets."acme-finished-standalone.com" = {
+        after = [ "acme-finished.example.com.service" "nginx.service" ];
+        wantedBy = [ "acme-finished.example.com.service" "nginx.service" ];
+      };
       systemd.services."acme-standalone.com" = {
-        wants = [ "acme-finished-standalone.com.target" ];
-        before = [ "acme-finished-standalone.com.target" ];
+        before = [ "nginx.service" ];
+        wantedBy = [ "nginx.service" ];
       };
       services.nginx.enable = true;
       services.nginx.virtualHosts."standalone.com" = {
@@ -71,12 +74,11 @@ in import ./make-test-python.nix {
 
       # A target remains active. Use this to probe the fact that
       # a service fired eventhough it is not RemainAfterExit
-      systemd.targets."acme-finished-a.example.com" = {};
-      systemd.services."acme-a.example.com" = {
-        wants = [ "acme-finished-a.example.com.target" ];
-        before = [ "acme-finished-a.example.com.target" ];
-        after = [ "nginx.service" ];
+      systemd.targets."acme-finished-a.example.com" = {
+        after = [ "acme-a.example.com.service" "nginx.service" ];
+        wantedBy = [ "acme-a.example.com.service" "nginx.service" ];
       };
+      systemd.services."acme-a.example.com" = {};
 
       services.nginx.enable = true;
 
@@ -92,12 +94,11 @@ in import ./make-test-python.nix {
       security.acme.server = "https://acme-v02.api.letsencrypt.org/dir";
 
       specialisation.second-cert.configuration = {pkgs, ...}: {
-        systemd.targets."acme-finished-b.example.com" = {};
-        systemd.services."acme-b.example.com" = {
-          wants = [ "acme-finished-b.example.com.target" ];
-          before = [ "acme-finished-b.example.com.target" ];
-          after = [ "nginx.service" ];
+        systemd.targets."acme-finished-b.example.com" = {
+          after = [ "acme-b.example.com.service" "nginx.service" ];
+          wantedBy = [ "acme-b.example.com.service" "nginx.service" ];
         };
+        systemd.services."acme-b.example.com" = {};
         services.nginx.virtualHosts."b.example.com" = {
           enableACME = true;
           forceSSL = true;
@@ -118,10 +119,12 @@ in import ./make-test-python.nix {
           user = config.services.nginx.user;
           group = config.services.nginx.group;
         };
-        systemd.targets."acme-finished-example.com" = {};
+        systemd.targets."acme-finished-example.com" = {
+          after = [ "acme-example.com.service" "nginx.service" ];
+          wantedBy = [ "acme-example.com.service" "nginx.service" ];
+        };
         systemd.services."acme-example.com" = {
-          wants = [ "acme-finished-example.com.target" ];
-          before = [ "acme-finished-example.com.target" "nginx.service" ];
+          before = [ "nginx.service" ];
           wantedBy = [ "nginx.service" ];
         };
         services.nginx.virtualHosts."c.example.com" = {

I think I saw a PR several days ago that heavily overlaps with this diff?

[emilazy]

Seems to have overlap with #85223.

[flokli]

Some of this might have been fixed by #85503.

[emilazy]

Sadly not, none of the test changes in that PR affect the dependency ordering :(

[stale]

I marked this as stale due to inactivity. → More info

[emilazy]

No idea if this still happens. cc @NixOS/acme

[m1cr0man]

Yeah I am pretty confident this is fixed. I went through a few iterations of systemd dependencies and awaiting async processes in the testing before realising the real issue was the non deterministic test certs. The issues with the JWS token corrupt are a different issue and something I will look into this weekend.