docker and group level network

[notnoop]

Nomad version

Nomad v0.12.0 (8f7fbc8e7b5a4ed0d0209968faf41b238e6d5817)

Operating system and Environment details

vagrant@linux:/opt/gopath/src/github.com/hashicorp/nomad$ cat /etc/os-release  | grep VERSION=
VERSION="18.04.4 LTS (Bionic Beaver)"
vagrant@linux:/opt/gopath/src/github.com/hashicorp/nomad$ uname -a
Linux linux 4.15.0-29-generic #31-Ubuntu SMP Tue Jul 17 15:39:52 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Networks eth0 and eth2:

$ ip addr show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 08:00:27:81:14:5e brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic eth0
       valid_lft 76021sec preferred_lft 76021sec
    inet6 fe80::a00:27ff:fe81:145e/64 scope link
       valid_lft forever preferred_lft forever
$ ip addr show eth1
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 08:00:27:2b:28:6c brd ff:ff:ff:ff:ff:ff
    inet 10.199.0.200/24 brd 10.199.0.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fe2b:286c/64 scope link
       valid_lft forever preferred_lft forever

Issue

When specifying network in the group level, and assign portings, docker container fail to start with Trying to map ports but no network interface is available.

Reproduction steps

1. Spin up a nomad agent configured with a host_network
2. submit the job
3. watch the allocation fail with "Trying to map ports but no network interface is available" error like in

$ nomad alloc status 7faaf70a
ID                     = 7faaf70a-ab1a-9d4a-dcfa-fa78f2b11305
Eval ID                = 2931e497
Name                   = lb-example.lb-example[0]
Node ID                = 6fcb0b06
Node Name              = nomad-devagent
Job ID                 = lb-example
Job Version            = 0
Client Status          = failed
Client Description     = Failed tasks
Desired Status         = run
Desired Description    = <none>
Created                = 5s ago
Modified               = 1s ago
Deployment ID          = 31ec75b9
Deployment Health      = unhealthy
Reschedule Eligibility = 25s from now

Allocation Addresses
Label  Dynamic  Address
*http  yes      10.199.0.200:80

Task "lb-example" is "dead"
Task Resources
CPU      Memory   Disk     Addresses
100 MHz  300 MiB  300 MiB

Task Events:
Started At     = N/A
Finished At    = 2020-07-21T23:39:48Z
Total Restarts = 0
Last Restart   = N/A

Recent Events:
Time                  Type             Description
2020-07-21T23:39:50Z  Killing          Sent interrupt. Waiting 5s before force killing
2020-07-21T23:39:48Z  Alloc Unhealthy  Unhealthy because of failed task
2020-07-21T23:39:48Z  Not Restarting   Error was unrecoverable
2020-07-21T23:39:48Z  Driver Failure   Failed to create container configuration for image "redis:3.2" ("sha256:87856cc39862cec77541d68382e4867d7ccb29a85a17221446c857ddaebca916"): Trying to map ports but no network interface is available
2020-07-21T23:39:48Z  Task Setup       Building Task Directory
2020-07-21T23:39:48Z  Received         Task received by client

Job file (if appropriate)

job "lb-example" {
  datacenters = "dc1"

  group "lb-example" {
    network {
      port "http" {
        static       = "80"
        host_network = "lb"
      }
    }

    task "lb-example" {
      driver = "docker"
      config {
        image = "redis:3.2"
        port_map {
          http = 80
        }

      }
    }
  }
}

Client config:

log_level = "TRACE"
enable_debug = true
datacenter = "dc1"
data_dir = "/tmp/nomad-data-dir"

client {
  enabled = true

  host_network "lb" {
    interface = "eth2"
  }

  servers = ["127.0.0.1:4647"]
  options = {
    "driver.raw_exec.enable" = "1"
    "docker.image.delay" = "10s"
  }
}

server {
  enabled          = true
  bootstrap_expect = 1
}

[Legogris]

Does the interface eth2 exist on Nomad startup? Your output is only showing eth0 and eth1.

[SapphicCode]

I don't know how relevant this is, but I just came across the same thing without the host_network stuff.

The former allocation fails with the same error, the latter works fine (but isn't documented anymore).

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.