network namespaces are not reconciled

[th0m]

Nomad version

Output from nomad version

$ nomad version
Nomad v1.1.4 (acd3d7889328ad1df2895eb714e2cbe3dd9c6d82)

Operating system and Environment details

Linux

Issue

Network namespaces associated to allocations that complete or fail while Nomad is stopped do not get cleaned up when Nomad starts back up and recovers the task.

This is a similar issue to #6385 but for network namespaces.

I am assuming this is due to https://github.com/hashicorp/nomad/blob/main/client/allocrunner/network_hook.go#L131-L140 not running when Nomad is stopped and the allocation completes.

Job file

sleep.nomad

job "sleep" {
  datacenters = ["dc1"]

  group "sleep-group" {
    network {
      mode = "bridge"
    }
    task "sleep-task" {
      driver = "exec"

      config {
        command = "/bin/sleep
        args    = ["60"]
      }

      resources {
        cpu    = 500
        memory = 256
      }
    }
  }
}

Reproduction steps

1. Run the job with nomad job run sleep.nomad
2. A network namespace gets created with the same id as the allocation

$ ip netns | grep 33a997ad
33a997ad-02f0-686b-8289-dcc9c4817711 (id: 4)

2. Stop nomad with systemctl stop nomad
3. Let the sleep 60 task complete
4. Observe the network namespace from the completed task is still there

$ ip netns | grep 33a997ad
33a997ad-02f0-686b-8289-dcc9c4817711 (id: 4)

4. Start nomad systemctl start nomad
5. Observe the network namespace from the completed task is still there

$ ip netns | grep 33a997ad
33a997ad-02f0-686b-8289-dcc9c4817711 (id: 4)

6. Observe Nomad created a new allocation for the job

$ nomad job status sleep
[...]
Allocations
ID        Node ID   Task Group   Version  Desired  Status   Created    Modified
e08c8f0e  c1d19991  sleep-group  0        run      running  30s ago    18s ago

5. And that a new associated network namespace has been created in addition to the one from the now defunct previous allocation.

$ ip netns | grep e08c8f0e
e08c8f0e-6a29-4b09-2900-2dda8acb2075 (id: 5)

Expected Result

I would expect the network namespaces from previous allocations to be regularly reconciled and garbage collected.

Actual Result

The network namespace associated to the now defunct 33a997ad allocation is still present on the machine

Feel free to let me know if you need any more information.
Thank you!

[jrasell]

Hi @th0m and thanks for another detailed issue! Reconciliation is something we should certainly add into the client so the network namespaces are cleaned as expected.

[th0m]

Hi @jrasell, I just noticed this was removed from "Needs Roadmapping", does this mean this will not be worked on internally?
Thank you!

[tgross]

Hi @th0m! The issue was moved to our internal "unified backlog" (which unfortunately isn't public) so that it can be picked up by a Nomad engineer to do. That being said, it's not on the near-term roadmap either.

[th0m]

Hi @tgross, sounds good, thanks for the quick response!

[th0m]

Adding some color to this issue, this is especially problematic when using the CNI vlan plugin.
There can only be a single vlan interface with the same parent interface and vlan id on a host (regardless of what network namespace they are part of).
Therefore, when such network namespaces leak, Nomad can't schedule allocations that want to use the CNI network with that same vlan id and parent interface on the host anymore.
This is essentially poisoning the host.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.