Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ubuntu image failing task with no logs on WSL2 #10983

Closed
TGNThump opened this issue Aug 1, 2021 · 2 comments
Closed

Ubuntu image failing task with no logs on WSL2 #10983

TGNThump opened this issue Aug 1, 2021 · 2 comments
Labels
stage/not-a-bug type/bug

Comments

@TGNThump
Copy link

TGNThump commented Aug 1, 2021

Nomad version

Nomad v1.1.3 (8c0c8140997329136971e66e4c2337dfcf932692)

Operating system and Environment details

Host is running Windows 10 21H1 (OS Build 19043.1110).
Nomad is running in -dev mode as root inside a Ubuntu (20.04.2) WSL2 (4.19.84-microsoft-standard) instance.
Docker Desktop v3.5.2 (66501), Docker Engine v20.10.7 is running on the host.

Issue

Attempting to deploy an ubuntu image job from the host using nomad job run .\ubuntu.jobspec.hcl results in the container failing to start.

Running an ubuntu image via docker run directly from both the host and WSL2 instance works as expected.

Reproduction steps

Run docker desktop with wsl2 integration enabled in linux containers mode and nomad server deployed in a WSL2 instance.

Expected Result

The ubuntu nomad job should start and the container should startup normally.

Actual Result

The container restarts repeatedly after the nomad reports the container exited with exit code zero.

Job file

job "ubuntu" {
  datacenters = ["dc1"]
  type = "service"
  group "ubuntu" {
    count = 1
    task "ubuntu" {
      driver = "docker"
      config {
        image = "ubuntu:latest"
      }
    }
  }
}

Docker logs

A docker desktop diagnostics zip has been set to [email protected]

Nomad logs

==> No configuration files loaded
==> Starting Nomad agent...
==> Nomad agent configuration:

       Advertise Addrs: HTTP: 192.168.21.129:4646; RPC: 192.168.21.129:4647; Serf: 192.168.21.129:4648
            Bind Addrs: HTTP: 0.0.0.0:4646; RPC: 0.0.0.0:4647; Serf: 0.0.0.0:4648
                Client: true
             Log Level: DEBUG
                Region: global (DC: dc1)
                Server: true
               Version: 1.1.3

==> Nomad agent started! Log data will stream in below:

    2021-08-01T15:44:56.275+0100 [DEBUG] agent.plugin_loader.docker: using client connection initialized from environment: plugin_dir=
    2021-08-01T15:44:56.276+0100 [DEBUG] agent.plugin_loader.docker: using client connection initialized from environment: plugin_dir=
    2021-08-01T15:44:56.276+0100 [INFO]  agent: detected plugin: name=java type=driver plugin_version=0.1.0
    2021-08-01T15:44:56.276+0100 [INFO]  agent: detected plugin: name=docker type=driver plugin_version=0.1.0
    2021-08-01T15:44:56.276+0100 [INFO]  agent: detected plugin: name=raw_exec type=driver plugin_version=0.1.0
    2021-08-01T15:44:56.276+0100 [INFO]  agent: detected plugin: name=exec type=driver plugin_version=0.1.0
    2021-08-01T15:44:56.276+0100 [INFO]  agent: detected plugin: name=qemu type=driver plugin_version=0.1.0
    2021-08-01T15:44:56.276+0100 [INFO]  agent: detected plugin: name=nvidia-gpu type=device plugin_version=0.1.0
    2021-08-01T15:44:56.277+0100 [INFO]  nomad.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:192.168.21.129:4647 Address:192.168.21.129:4647}]"
    2021-08-01T15:44:56.277+0100 [INFO]  nomad.raft: entering follower state: follower="Node at 192.168.21.129:4647 [Follower]" leader=
    2021-08-01T15:44:56.277+0100 [INFO]  nomad: serf: EventMemberJoin: BPPC.global 192.168.21.129
    2021-08-01T15:44:56.277+0100 [INFO]  nomad: starting scheduling worker(s): num_workers=4 schedulers=[batch, system, service, _core]
    2021-08-01T15:44:56.277+0100 [INFO]  nomad: adding server: server="BPPC.global (Addr: 192.168.21.129:4647) (DC: dc1)"
    2021-08-01T15:44:56.277+0100 [INFO]  client: using state directory: state_dir=/tmp/NomadClient706558449
    2021-08-01T15:44:56.277+0100 [INFO]  client: using alloc directory: alloc_dir=/tmp/NomadClient451030172
    2021-08-01T15:44:56.278+0100 [INFO]  client.cpuset_manager: initialized cpuset cgroup manager: parent=/nomad cpuset=0-3
    2021-08-01T15:44:56.278+0100 [DEBUG] client.fingerprint_mgr: built-in fingerprints: fingerprinters=[arch, bridge, cgroup, cni, consul, cpu, host, memory, network, nomad, signal, storage, vault, env_aws, env_gce, env_azure]
    2021-08-01T15:44:56.279+0100 [WARN]  client.fingerprint_mgr: failed to detect bridge kernel module, bridge network mode disabled: error="3 errors occurred:
	* module bridge not in /proc/modules
	* failed to open /lib/modules/4.19.84-microsoft-standard/modules.builtin: open /lib/modules/4.19.84-microsoft-standard/modules.builtin: no such file or directory
	* failed to open /lib/modules/4.19.84-microsoft-standard/modules.dep: open /lib/modules/4.19.84-microsoft-standard/modules.dep: no such file or directory

"
    2021-08-01T15:44:56.279+0100 [INFO]  client.fingerprint_mgr.cgroup: cgroups are available
    2021-08-01T15:44:56.279+0100 [DEBUG] client.fingerprint_mgr: CNI config dir is not set or does not exist, skipping: cni_config_dir=/opt/cni/config
    2021-08-01T15:44:56.279+0100 [DEBUG] client.fingerprint_mgr: fingerprinting periodically: fingerprinter=cgroup period=15s
    2021-08-01T15:44:56.279+0100 [DEBUG] client.fingerprint_mgr: fingerprinting periodically: fingerprinter=consul period=15s
    2021-08-01T15:44:56.280+0100 [DEBUG] client.fingerprint_mgr.cpu: detected cpu frequency: MHz=3503
    2021-08-01T15:44:56.280+0100 [DEBUG] client.fingerprint_mgr.cpu: detected core count: cores=4
    2021-08-01T15:44:56.280+0100 [DEBUG] client.fingerprint_mgr.cpu: detected reservable cores: cpuset=[0, 1, 2, 3]
    2021-08-01T15:44:56.281+0100 [WARN]  client.fingerprint_mgr.network: unable to parse speed: path=/usr/sbin/ethtool device=lo
    2021-08-01T15:44:56.281+0100 [DEBUG] client.fingerprint_mgr.network: unable to read link speed: path=/sys/class/net/lo/speed
    2021-08-01T15:44:56.281+0100 [DEBUG] client.fingerprint_mgr.network: link speed could not be detected and no speed specified by user, falling back to default speed: mbits=1000
    2021-08-01T15:44:56.281+0100 [DEBUG] client.fingerprint_mgr.network: detected interface IP: interface=lo IP=127.0.0.1
    2021-08-01T15:44:56.281+0100 [DEBUG] client.fingerprint_mgr.network: detected interface IP: interface=lo IP=::1
    2021-08-01T15:44:56.282+0100 [WARN]  client.fingerprint_mgr.network: unable to parse speed: path=/usr/sbin/ethtool device=lo
    2021-08-01T15:44:56.282+0100 [DEBUG] client.fingerprint_mgr.network: unable to read link speed: path=/sys/class/net/lo/speed
    2021-08-01T15:44:56.282+0100 [DEBUG] client.fingerprint_mgr.network: link speed could not be detected, falling back to default speed: mbits=1000
    2021-08-01T15:44:56.284+0100 [WARN]  client.fingerprint_mgr.network: unable to parse speed: path=/usr/sbin/ethtool device=bond0
    2021-08-01T15:44:56.284+0100 [DEBUG] client.fingerprint_mgr.network: unable to read link speed: path=/sys/class/net/bond0/speed
    2021-08-01T15:44:56.284+0100 [DEBUG] client.fingerprint_mgr.network: link speed could not be detected, falling back to default speed: mbits=1000
    2021-08-01T15:44:56.284+0100 [WARN]  client.fingerprint_mgr.network: error calling ethtool: error="exit status 75" path=/usr/sbin/ethtool device=dummy0
    2021-08-01T15:44:56.284+0100 [DEBUG] client.fingerprint_mgr.network: unable to read link speed: path=/sys/class/net/dummy0/speed
    2021-08-01T15:44:56.284+0100 [DEBUG] client.fingerprint_mgr.network: link speed could not be detected, falling back to default speed: mbits=1000
    2021-08-01T15:44:56.286+0100 [WARN]  client.fingerprint_mgr.network: error calling ethtool: error="exit status 75" path=/usr/sbin/ethtool device=sit0
    2021-08-01T15:44:56.286+0100 [DEBUG] client.fingerprint_mgr.network: unable to read link speed: path=/sys/class/net/sit0/speed
    2021-08-01T15:44:56.286+0100 [DEBUG] client.fingerprint_mgr.network: link speed could not be detected, falling back to default speed: mbits=1000
    2021-08-01T15:44:56.287+0100 [DEBUG] client.fingerprint_mgr: fingerprinting periodically: fingerprinter=vault period=15s
    2021-08-01T15:44:57.393+0100 [WARN]  nomad.raft: heartbeat timeout reached, starting election: last-leader=
    2021-08-01T15:44:57.393+0100 [INFO]  nomad.raft: entering candidate state: node="Node at 192.168.21.129:4647 [Candidate]" term=2
    2021-08-01T15:44:57.393+0100 [DEBUG] nomad.raft: votes: needed=1
    2021-08-01T15:44:57.393+0100 [DEBUG] nomad.raft: vote granted: from=192.168.21.129:4647 term=2 tally=1
    2021-08-01T15:44:57.393+0100 [INFO]  nomad.raft: election won: tally=1
    2021-08-01T15:44:57.393+0100 [INFO]  nomad.raft: entering leader state: leader="Node at 192.168.21.129:4647 [Leader]"
    2021-08-01T15:44:57.393+0100 [INFO]  nomad: cluster leadership acquired
    2021-08-01T15:44:57.394+0100 [INFO]  nomad.core: established cluster id: cluster_id=05e1c2a7-f8c8-8f89-2479-dccc22209772 create_time=1627829097393950400
    2021-08-01T15:45:02.289+0100 [DEBUG] client.fingerprint_mgr.env_gce: could not read value for attribute: attribute=machine-type error="Get "http://169.254.169.254/computeMetadata/v1/instance/machine-type": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
    2021-08-01T15:45:02.289+0100 [DEBUG] client.fingerprint_mgr.env_gce: error querying GCE Metadata URL, skipping
    2021-08-01T15:45:04.290+0100 [DEBUG] client.fingerprint_mgr.env_azure: could not read value for attribute: attribute=compute/azEnvironment error="Get "http://169.254.169.254/metadata/instance/compute/azEnvironment?api-version=2019-06-04&format=text": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
    2021-08-01T15:45:04.290+0100 [DEBUG] client.fingerprint_mgr: detected fingerprints: node_attrs=[arch, cgroup, cpu, host, network, nomad, signal, storage]
    2021-08-01T15:45:04.290+0100 [INFO]  client.plugin: starting plugin manager: plugin-type=csi
    2021-08-01T15:45:04.290+0100 [INFO]  client.plugin: starting plugin manager: plugin-type=driver
    2021-08-01T15:45:04.290+0100 [INFO]  client.plugin: starting plugin manager: plugin-type=device
    2021-08-01T15:45:04.290+0100 [DEBUG] client.driver_mgr: initial driver fingerprint: driver=qemu health=undetected description=
    2021-08-01T15:45:04.290+0100 [DEBUG] client.plugin: waiting on plugin manager initial fingerprint: plugin-type=driver
    2021-08-01T15:45:04.290+0100 [DEBUG] client.plugin: waiting on plugin manager initial fingerprint: plugin-type=device
    2021-08-01T15:45:04.290+0100 [DEBUG] client.driver_mgr: initial driver fingerprint: driver=exec health=healthy description=Healthy
    2021-08-01T15:45:04.290+0100 [DEBUG] client.driver_mgr: initial driver fingerprint: driver=raw_exec health=healthy description=Healthy
    2021-08-01T15:45:04.290+0100 [DEBUG] client.plugin: finished plugin manager initial fingerprint: plugin-type=device
    2021-08-01T15:45:04.291+0100 [DEBUG] client.server_mgr: new server list: new_servers=[0.0.0.0:4647, 192.168.21.129:4647] old_servers=[]
    2021-08-01T15:45:04.313+0100 [DEBUG] client.driver_mgr: initial driver fingerprint: driver=docker health=healthy description=Healthy
    2021-08-01T15:45:04.328+0100 [DEBUG] client.driver_mgr: initial driver fingerprint: driver=java health=healthy description=Healthy
    2021-08-01T15:45:04.328+0100 [DEBUG] client.driver_mgr: detected drivers: drivers="map[healthy:[exec raw_exec docker java] undetected:[qemu]]"
    2021-08-01T15:45:04.328+0100 [DEBUG] client.plugin: finished plugin manager initial fingerprint: plugin-type=driver
    2021-08-01T15:45:04.328+0100 [INFO]  client: started client: node_id=e2b13088-68e7-b724-639d-468cb36af06b
    2021-08-01T15:45:04.330+0100 [DEBUG] client: updated allocations: index=1 total=0 pulled=0 filtered=0
    2021-08-01T15:45:04.330+0100 [DEBUG] client: allocation updates: added=0 removed=0 updated=0 ignored=0
    2021-08-01T15:45:04.330+0100 [DEBUG] client: allocation updates applied: added=0 removed=0 updated=0 ignored=0 errors=0
    2021-08-01T15:45:04.330+0100 [INFO]  client: node registration complete
    2021-08-01T15:45:04.331+0100 [DEBUG] client: state updated: node_status=ready
    2021-08-01T15:45:04.331+0100 [DEBUG] client.server_mgr: new server list: new_servers=[192.168.21.129:4647] old_servers=[192.168.21.129:4647, 0.0.0.0:4647]
    2021-08-01T15:45:05.330+0100 [DEBUG] client: state changed, updating node and re-registering
    2021-08-01T15:45:05.331+0100 [INFO]  client: node registration complete
    2021-08-01T15:45:19.981+0100 [DEBUG] worker: dequeued evaluation: eval_id=e7d6cce4-8b0f-7834-e169-485e4b0c2a65
    2021-08-01T15:45:19.981+0100 [DEBUG] http: request complete: method=PUT path=/v1/jobs duration=2.3068ms
    2021-08-01T15:45:19.981+0100 [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=e7d6cce4-8b0f-7834-e169-485e4b0c2a65 job_id=ubuntu namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 0)
Created Deployment: "cf1f5873-90b8-7016-e67a-8e32a3d4d895"
Desired Changes for "ubuntu": (place 1) (inplace 0) (destructive 0) (stop 0) (migrate 0) (ignore 0) (canary 0)"
    2021-08-01T15:45:19.982+0100 [DEBUG] worker: submitted plan for evaluation: eval_id=e7d6cce4-8b0f-7834-e169-485e4b0c2a65
    2021-08-01T15:45:19.982+0100 [DEBUG] worker.service_sched: setting eval status: eval_id=e7d6cce4-8b0f-7834-e169-485e4b0c2a65 job_id=ubuntu namespace=default status=complete
    2021-08-01T15:45:19.982+0100 [DEBUG] worker: updated evaluation: eval="<Eval "e7d6cce4-8b0f-7834-e169-485e4b0c2a65" JobID: "ubuntu" Namespace: "default">"
    2021-08-01T15:45:19.982+0100 [DEBUG] worker: ack evaluation: eval_id=e7d6cce4-8b0f-7834-e169-485e4b0c2a65
    2021-08-01T15:45:19.983+0100 [DEBUG] client: updated allocations: index=11 total=1 pulled=1 filtered=0
    2021-08-01T15:45:19.983+0100 [DEBUG] client: allocation updates: added=1 removed=0 updated=0 ignored=0
    2021-08-01T15:45:19.983+0100 [DEBUG] client: allocation updates applied: added=1 removed=0 updated=0 ignored=0 errors=0
    2021-08-01T15:45:19.984+0100 [DEBUG] client.alloc_runner.task_runner: lifecycle start condition has been met, proceeding: alloc_id=099b416d-3535-8306-ba70-aca746be83af task=ubuntu
    2021-08-01T15:45:19.984+0100 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: starting plugin: alloc_id=099b416d-3535-8306-ba70-aca746be83af task=ubuntu path=/usr/bin/nomad args=[/usr/bin/nomad, logmon]
    2021-08-01T15:45:19.984+0100 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: plugin started: alloc_id=099b416d-3535-8306-ba70-aca746be83af task=ubuntu path=/usr/bin/nomad pid=19672
    2021-08-01T15:45:19.984+0100 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: waiting for RPC address: alloc_id=099b416d-3535-8306-ba70-aca746be83af task=ubuntu path=/usr/bin/nomad
    2021-08-01T15:45:19.988+0100 [DEBUG] http: request complete: method=GET path=/v1/evaluation/e7d6cce4-8b0f-7834-e169-485e4b0c2a65 duration=253.2µs
    2021-08-01T15:45:19.996+0100 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon.nomad: plugin address: alloc_id=099b416d-3535-8306-ba70-aca746be83af task=ubuntu @module=logmon address=/tmp/plugin155863015 network=unix timestamp=2021-08-01T15:45:19.996+0100
    2021-08-01T15:45:19.996+0100 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: using plugin: alloc_id=099b416d-3535-8306-ba70-aca746be83af task=ubuntu version=2
    2021-08-01T15:45:19.997+0100 [DEBUG] http: request complete: method=GET path=/v1/evaluation/e7d6cce4-8b0f-7834-e169-485e4b0c2a65/allocations duration=227.8µs
    2021-08-01T15:45:20.003+0100 [INFO]  client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=099b416d-3535-8306-ba70-aca746be83af task=ubuntu path=/tmp/NomadClient451030172/099b416d-3535-8306-ba70-aca746be83af/alloc/logs/.ubuntu.stdout.fifo @module=logmon timestamp=2021-08-01T15:45:20.002+0100
    2021-08-01T15:45:20.003+0100 [INFO]  client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=099b416d-3535-8306-ba70-aca746be83af task=ubuntu @module=logmon path=/tmp/NomadClient451030172/099b416d-3535-8306-ba70-aca746be83af/alloc/logs/.ubuntu.stderr.fifo timestamp=2021-08-01T15:45:20.002+0100
    2021-08-01T15:45:20.180+0100 [DEBUG] client: updated allocations: index=13 total=1 pulled=0 filtered=1
    2021-08-01T15:45:20.180+0100 [DEBUG] client: allocation updates: added=0 removed=0 updated=0 ignored=1
    2021-08-01T15:45:20.180+0100 [DEBUG] client: allocation updates applied: added=0 removed=0 updated=0 ignored=1 errors=0
    2021-08-01T15:45:24.587+0100 [DEBUG] client.driver_mgr.docker: docker pull succeeded: driver=docker image_ref=ubuntu:latest
    2021-08-01T15:45:24.591+0100 [DEBUG] client.driver_mgr.docker: image reference count incremented: driver=docker image_name=ubuntu:latest image_id=sha256:1318b700e415001198d1bf66d260b07f67ca8a552b61b0da02b3832c778f221b references=1
    2021-08-01T15:45:24.591+0100 [DEBUG] client.driver_mgr.docker: configured resources: driver=docker task_name=ubuntu memory=314572800 memory_reservation=0 cpu_shares=100 cpu_quota=0 cpu_period=0
    2021-08-01T15:45:24.591+0100 [DEBUG] client.driver_mgr.docker: binding directories: driver=docker task_name=ubuntu binds="[]string{"/tmp/NomadClient451030172/099b416d-3535-8306-ba70-aca746be83af/alloc:/alloc", "/tmp/NomadClient451030172/099b416d-3535-8306-ba70-aca746be83af/ubuntu/local:/local", "/tmp/NomadClient451030172/099b416d-3535-8306-ba70-aca746be83af/ubuntu/secrets:/secrets"}"
    2021-08-01T15:45:24.591+0100 [DEBUG] client.driver_mgr.docker: networking mode not specified; using default: driver=docker task_name=ubuntu
    2021-08-01T15:45:24.591+0100 [DEBUG] client.driver_mgr.docker: applied labels on the container: driver=docker task_name=ubuntu labels=map[com.hashicorp.nomad.alloc_id:099b416d-3535-8306-ba70-aca746be83af]
    2021-08-01T15:45:24.591+0100 [DEBUG] client.driver_mgr.docker: setting container name: driver=docker task_name=ubuntu container_name=ubuntu-099b416d-3535-8306-ba70-aca746be83af
    2021-08-01T15:45:24.747+0100 [INFO]  client.driver_mgr.docker: created container: driver=docker container_id=7fab3ef88a161b01ddea5b01dd30bf02407dda70d6331578b18b64783fe53564
    2021-08-01T15:45:25.774+0100 [INFO]  client.driver_mgr.docker: started container: driver=docker container_id=7fab3ef88a161b01ddea5b01dd30bf02407dda70d6331578b18b64783fe53564
    2021-08-01T15:45:25.774+0100 [DEBUG] client.driver_mgr.docker.docker_logger: starting plugin: driver=docker path=/usr/bin/nomad args=[/usr/bin/nomad, docker_logger]
    2021-08-01T15:45:25.775+0100 [DEBUG] client.driver_mgr.docker.docker_logger: plugin started: driver=docker path=/usr/bin/nomad pid=19694
    2021-08-01T15:45:25.775+0100 [DEBUG] client.driver_mgr.docker.docker_logger: waiting for RPC address: driver=docker path=/usr/bin/nomad
    2021-08-01T15:45:25.790+0100 [DEBUG] client.driver_mgr.docker.docker_logger.nomad: plugin address: driver=docker @module=docker_logger address=/tmp/plugin238764221 network=unix timestamp=2021-08-01T15:45:25.789+0100
    2021-08-01T15:45:25.790+0100 [DEBUG] client.driver_mgr.docker.docker_logger: using plugin: driver=docker version=2
    2021-08-01T15:45:25.802+0100 [DEBUG] client.driver_mgr.docker.docker_logger.nomad: using client connection initialized from environment: driver=docker @module=docker_logger timestamp=2021-08-01T15:45:25.800+0100
    2021-08-01T15:45:25.879+0100 [DEBUG] client.driver_mgr.docker.docker_logger.stdio: received EOF, stopping recv loop: driver=docker err="rpc error: code = Unavailable desc = transport is closing"
    2021-08-01T15:45:25.880+0100 [DEBUG] client.driver_mgr.docker.docker_logger: plugin process exited: driver=docker path=/usr/bin/nomad pid=19694
    2021-08-01T15:45:25.880+0100 [DEBUG] client.driver_mgr.docker.docker_logger: plugin exited: driver=docker
    2021-08-01T15:45:25.966+0100 [DEBUG] client.driver_mgr.docker: image id reference count decremented: driver=docker image_id=sha256:1318b700e415001198d1bf66d260b07f67ca8a552b61b0da02b3832c778f221b references=0
    2021-08-01T15:45:25.966+0100 [INFO]  client.alloc_runner.task_runner: restarting task: alloc_id=099b416d-3535-8306-ba70-aca746be83af task=ubuntu reason="Restart within policy" delay=18.0358003s
    2021-08-01T15:45:26.035+0100 [DEBUG] client: updated allocations: index=14 total=1 pulled=0 filtered=1
    2021-08-01T15:45:26.035+0100 [DEBUG] client: allocation updates: added=0 removed=0 updated=0 ignored=1
    2021-08-01T15:45:26.035+0100 [DEBUG] client: allocation updates applied: added=0 removed=0 updated=0 ignored=1 errors=0
    2021-08-01T15:45:44.003+0100 [INFO]  client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=099b416d-3535-8306-ba70-aca746be83af task=ubuntu path=/tmp/NomadClient451030172/099b416d-3535-8306-ba70-aca746be83af/alloc/logs/.ubuntu.stdout.fifo @module=logmon timestamp=2021-08-01T15:45:44.003+0100
    2021-08-01T15:45:44.003+0100 [INFO]  client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=099b416d-3535-8306-ba70-aca746be83af task=ubuntu @module=logmon path=/tmp/NomadClient451030172/099b416d-3535-8306-ba70-aca746be83af/alloc/logs/.ubuntu.stderr.fifo timestamp=2021-08-01T15:45:44.003+0100
    2021-08-01T15:45:44.087+0100 [DEBUG] client: updated allocations: index=15 total=1 pulled=0 filtered=1
    2021-08-01T15:45:44.087+0100 [DEBUG] client: allocation updates: added=0 removed=0 updated=0 ignored=1
    2021-08-01T15:45:44.087+0100 [DEBUG] client: allocation updates applied: added=0 removed=0 updated=0 ignored=1 errors=0
    2021-08-01T15:45:48.604+0100 [DEBUG] client.driver_mgr.docker: docker pull succeeded: driver=docker image_ref=ubuntu:latest
    2021-08-01T15:45:48.608+0100 [DEBUG] client.driver_mgr.docker: cancelling removal of container image: driver=docker image_name=ubuntu:latest
    2021-08-01T15:45:48.608+0100 [DEBUG] client.driver_mgr.docker: image reference count incremented: driver=docker image_name=ubuntu:latest image_id=sha256:1318b700e415001198d1bf66d260b07f67ca8a552b61b0da02b3832c778f221b references=1
    2021-08-01T15:45:48.608+0100 [DEBUG] client.driver_mgr.docker: configured resources: driver=docker task_name=ubuntu memory=314572800 memory_reservation=0 cpu_shares=100 cpu_quota=0 cpu_period=0
    2021-08-01T15:45:48.608+0100 [DEBUG] client.driver_mgr.docker: binding directories: driver=docker task_name=ubuntu binds="[]string{"/tmp/NomadClient451030172/099b416d-3535-8306-ba70-aca746be83af/alloc:/alloc", "/tmp/NomadClient451030172/099b416d-3535-8306-ba70-aca746be83af/ubuntu/local:/local", "/tmp/NomadClient451030172/099b416d-3535-8306-ba70-aca746be83af/ubuntu/secrets:/secrets"}"
    2021-08-01T15:45:48.608+0100 [DEBUG] client.driver_mgr.docker: networking mode not specified; using default: driver=docker task_name=ubuntu
    2021-08-01T15:45:48.609+0100 [DEBUG] client.driver_mgr.docker: applied labels on the container: driver=docker task_name=ubuntu labels=map[com.hashicorp.nomad.alloc_id:099b416d-3535-8306-ba70-aca746be83af]
    2021-08-01T15:45:48.609+0100 [DEBUG] client.driver_mgr.docker: setting container name: driver=docker task_name=ubuntu container_name=ubuntu-099b416d-3535-8306-ba70-aca746be83af
    2021-08-01T15:45:48.770+0100 [INFO]  client.driver_mgr.docker: created container: driver=docker container_id=98a13312650f6bdfb5376e91306acec6abb356ff9472a4795fb0aa458f509d45
    2021-08-01T15:45:49.549+0100 [INFO]  client.driver_mgr.docker: started container: driver=docker container_id=98a13312650f6bdfb5376e91306acec6abb356ff9472a4795fb0aa458f509d45
    2021-08-01T15:45:49.549+0100 [DEBUG] client.driver_mgr.docker.docker_logger: starting plugin: driver=docker path=/usr/bin/nomad args=[/usr/bin/nomad, docker_logger]
    2021-08-01T15:45:49.550+0100 [DEBUG] client.driver_mgr.docker.docker_logger: plugin started: driver=docker path=/usr/bin/nomad pid=19768
    2021-08-01T15:45:49.550+0100 [DEBUG] client.driver_mgr.docker.docker_logger: waiting for RPC address: driver=docker path=/usr/bin/nomad
    2021-08-01T15:45:49.557+0100 [DEBUG] client.driver_mgr.docker.docker_logger: using plugin: driver=docker version=2
    2021-08-01T15:45:49.557+0100 [DEBUG] client.driver_mgr.docker.docker_logger.nomad: plugin address: driver=docker address=/tmp/plugin339766319 network=unix @module=docker_logger timestamp=2021-08-01T15:45:49.557+0100
    2021-08-01T15:45:49.558+0100 [DEBUG] client.driver_mgr.docker.docker_logger.nomad: using client connection initialized from environment: driver=docker @module=docker_logger timestamp=2021-08-01T15:45:49.558+0100
    2021-08-01T15:45:49.617+0100 [DEBUG] client.driver_mgr.docker.docker_logger.stdio: received EOF, stopping recv loop: driver=docker err="rpc error: code = Unavailable desc = transport is closing"
    2021-08-01T15:45:49.618+0100 [DEBUG] client.driver_mgr.docker.docker_logger: plugin process exited: driver=docker path=/usr/bin/nomad pid=19768
    2021-08-01T15:45:49.618+0100 [DEBUG] client.driver_mgr.docker.docker_logger: plugin exited: driver=docker
    2021-08-01T15:45:49.693+0100 [DEBUG] client.driver_mgr.docker: image id reference count decremented: driver=docker image_id=sha256:1318b700e415001198d1bf66d260b07f67ca8a552b61b0da02b3832c778f221b references=0
    2021-08-01T15:45:49.693+0100 [INFO]  client.alloc_runner.task_runner: restarting task: alloc_id=099b416d-3535-8306-ba70-aca746be83af task=ubuntu reason="Restart within policy" delay=16.089544144s
    2021-08-01T15:45:49.738+0100 [DEBUG] client: updated allocations: index=16 total=1 pulled=0 filtered=1
    2021-08-01T15:45:49.738+0100 [DEBUG] client: allocation updates: added=0 removed=0 updated=0 ignored=1
    2021-08-01T15:45:49.738+0100 [DEBUG] client: allocation updates applied: added=0 removed=0 updated=0 ignored=1 errors=0
    2021-08-01T15:45:49.990+0100 [DEBUG] client: updated allocations: index=17 total=1 pulled=0 filtered=1
    2021-08-01T15:45:49.990+0100 [DEBUG] client: allocation updates: added=0 removed=0 updated=0 ignored=1
    2021-08-01T15:45:49.990+0100 [DEBUG] client: allocation updates applied: added=0 removed=0 updated=0 ignored=1 errors=0
@lgfa29
Copy link
Contributor

lgfa29 commented Aug 19, 2021

Hi @TGNThump.

Is the jobfile you sent what you are trying to run?

If so, the image used in the job doesn't actually do anything except calling bash, which just exits in a non-interactive environment, like Nomad.

If you want the container to hang around you will need to have it do something. Here's an example that just keeps a loop going:

job "ubuntu" {
  datacenters = ["dc1"]
  type        = "service"
  group "ubuntu" {
    count = 1
    task "ubuntu" {
      driver = "docker"
      config {
        image   = "ubuntu:latest"
        command = "bash"
        args    = ["-c", "while true; do sleep 10; done"]
      }
    }
  }
}

I'm going to close this issue, but let me know how it goes 🙂

@lgfa29 lgfa29 closed this as completed Aug 19, 2021
@tgross tgross mentioned this issue Nov 8, 2021
Open
@github-actions
Copy link

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 16, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
stage/not-a-bug type/bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@lgfa29 @TGNThump