2020-05-28T16:37:49.903Z [WARN]  agent.plugin_loader: skipping external plugins since plugin_dir doesn't exist: plugin_dir=/opt/nomad/plugins
2020-05-28T16:37:49.904Z [DEBUG] agent.plugin_loader.docker: using client connection initialized from environment: plugin_dir=/opt/nomad/plugins
2020-05-28T16:37:49.904Z [DEBUG] agent.plugin_loader.docker: using client connection initialized from environment: plugin_dir=/opt/nomad/plugins
2020-05-28T16:37:49.904Z [INFO]  agent: detected plugin: name=exec type=driver plugin_version=0.1.0
2020-05-28T16:37:49.904Z [INFO]  agent: detected plugin: name=qemu type=driver plugin_version=0.1.0
2020-05-28T16:37:49.904Z [INFO]  agent: detected plugin: name=java type=driver plugin_version=0.1.0
2020-05-28T16:37:49.904Z [INFO]  agent: detected plugin: name=docker type=driver plugin_version=0.1.0
2020-05-28T16:37:49.904Z [INFO]  agent: detected plugin: name=raw_exec type=driver plugin_version=0.1.0
2020-05-28T16:37:49.904Z [INFO]  agent: detected plugin: name=nvidia-gpu type=device plugin_version=0.1.0
2020-05-28T16:37:49.909Z [INFO]  nomad.raft: initial configuration: index=265 servers="[{Suffrage:Voter ID:192.168.50.92:4647 Address:192.168.50.92:4647} {Suffrage:Voter ID:192.168.50.93:4647 Address:192.168.50.93:4647} {Suffrage:Voter ID:192.168.50.91:4647 Address:192.168.50.91:4647}]"
2020-05-28T16:37:49.910Z [INFO]  nomad: serf: EventMemberJoin: three-photon.global 192.168.50.93
2020-05-28T16:37:49.910Z [INFO]  nomad: starting scheduling worker(s): num_workers=2 schedulers=[service, batch, system, _core]
2020-05-28T16:37:49.910Z [INFO]  client: using state directory: state_dir=/opt/nomad/client
2020-05-28T16:37:49.910Z [INFO]  client: using alloc directory: alloc_dir=/opt/nomad/alloc
2020-05-28T16:37:49.911Z [DEBUG] client.fingerprint_mgr: built-in fingerprints: fingerprinters=[arch, cgroup, consul, cpu, host, memory, network, nomad, signal, storage, vault, env_gce, env_aws]
2020-05-28T16:37:49.911Z [INFO]  client.fingerprint_mgr.cgroup: cgroups are available
2020-05-28T16:37:49.913Z [INFO]  nomad.raft: entering follower state: follower="Node at 192.168.50.93:4647 [Follower]" leader=
2020-05-28T16:37:49.913Z [INFO]  nomad: serf: Attempting re-join to previously known node: two-photon.global: 192.168.50.92:4648
2020-05-28T16:37:49.913Z [DEBUG] nomad: memberlist: Initiating push/pull sync with: 192.168.50.92:4648
2020-05-28T16:37:49.914Z [INFO]  nomad: adding server: server="three-photon.global (Addr: 192.168.50.93:4647) (DC: dev-setup)"
2020-05-28T16:37:49.914Z [DEBUG] client.fingerprint_mgr: fingerprinting periodically: fingerprinter=cgroup period=15s
2020-05-28T16:37:49.916Z [INFO]  client.fingerprint_mgr.consul: consul agent is available
2020-05-28T16:37:49.916Z [DEBUG] client.fingerprint_mgr.cpu: detected cpu frequency: MHz=2399
2020-05-28T16:37:49.916Z [DEBUG] client.fingerprint_mgr.cpu: detected core count: cores=2
2020-05-28T16:37:49.917Z [DEBUG] client.fingerprint_mgr: fingerprinting periodically: fingerprinter=consul period=15s
2020-05-28T16:37:49.917Z [DEBUG] client.fingerprint_mgr.network: link speed detected: interface=eth1 mbits=1000
2020-05-28T16:37:49.917Z [DEBUG] client.fingerprint_mgr.network: detected interface IP: interface=eth1 IP=192.168.50.93
2020-05-28T16:37:49.919Z [WARN]  nomad: memberlist: Refuting an alive message
2020-05-28T16:37:49.920Z [INFO]  nomad: serf: EventMemberJoin: one-photon.global 192.168.50.91
2020-05-28T16:37:49.920Z [INFO]  nomad: serf: EventMemberJoin: two-photon.global 192.168.50.92
2020-05-28T16:37:49.920Z [INFO]  nomad: serf: Re-joined to previously known node: two-photon.global: 192.168.50.92:4648
2020-05-28T16:37:49.920Z [INFO]  nomad: adding server: server="one-photon.global (Addr: 192.168.50.91:4647) (DC: dev-setup)"
2020-05-28T16:37:49.920Z [INFO]  nomad: adding server: server="two-photon.global (Addr: 192.168.50.92:4647) (DC: dev-setup)"
2020-05-28T16:37:49.926Z [INFO]  client.fingerprint_mgr.vault: Vault is available
2020-05-28T16:37:49.926Z [DEBUG] client.fingerprint_mgr: fingerprinting periodically: fingerprinter=vault period=15s
2020-05-28T16:37:49.928Z [DEBUG] client.fingerprint_mgr.env_gce: could not read value for attribute: attribute=machine-type error="Get "http://169.254.169.254/computeMetadata/v1/instance/machine-type": dial tcp 169.254.169.254:80: connect: network is unreachable"
2020-05-28T16:37:49.928Z [DEBUG] client.fingerprint_mgr.env_gce: error querying GCE Metadata URL, skipping
2020-05-28T16:37:49.928Z [DEBUG] client.fingerprint_mgr: detected fingerprints: node_attrs=[arch, cgroup, consul, cpu, host, network, nomad, signal, storage, vault]
2020-05-28T16:37:49.928Z [INFO]  client.plugin: starting plugin manager: plugin-type=csi
2020-05-28T16:37:49.928Z [INFO]  client.plugin: starting plugin manager: plugin-type=driver
2020-05-28T16:37:49.928Z [INFO]  client.plugin: starting plugin manager: plugin-type=device
2020-05-28T16:37:49.929Z [DEBUG] client.driver_mgr: initial driver fingerprint: driver=raw_exec health=healthy description=Healthy
2020-05-28T16:37:49.929Z [DEBUG] client.driver_mgr: initial driver fingerprint: driver=exec health=healthy description=Healthy
2020-05-28T16:37:49.929Z [DEBUG] client.driver_mgr: initial driver fingerprint: driver=qemu health=undetected description=
2020-05-28T16:37:49.930Z [DEBUG] client.driver_mgr: initial driver fingerprint: driver=java health=undetected description=
2020-05-28T16:37:49.930Z [DEBUG] nomad.vault: starting renewal loop: creation_ttl=72h0m0s
2020-05-28T16:37:49.931Z [DEBUG] client.plugin: waiting on plugin manager initial fingerprint: plugin-type=driver
2020-05-28T16:37:49.931Z [DEBUG] client.plugin: waiting on plugin manager initial fingerprint: plugin-type=device
2020-05-28T16:37:49.931Z [DEBUG] client.plugin: finished plugin manager initial fingerprint: plugin-type=device
2020-05-28T16:37:49.932Z [DEBUG] client.server_mgr: new server list: new_servers=[0.0.0.0:4647, 192.168.50.93:4647] old_servers=[]
2020-05-28T16:37:49.942Z [DEBUG] client.driver_mgr: initial driver fingerprint: driver=docker health=healthy description=Healthy
2020-05-28T16:37:49.949Z [DEBUG] client.driver_mgr: detected drivers: drivers="map[healthy:[raw_exec exec docker] undetected:[qemu java]]"
2020-05-28T16:37:49.949Z [DEBUG] client.plugin: finished plugin manager initial fingerprint: plugin-type=driver
2020-05-28T16:37:49.950Z [INFO]  client: started client: node_id=8cd0bef0-a6ac-a721-da26-cc9975bc2fc9
2020-05-28T16:37:49.950Z [DEBUG] client: updated allocations: index=1 total=0 pulled=0 filtered=0
2020-05-28T16:37:49.950Z [DEBUG] client: allocation updates: added=0 removed=0 updated=0 ignored=0
2020-05-28T16:37:49.950Z [DEBUG] client: allocation updates applied: added=0 removed=0 updated=0 ignored=0 errors=0
2020-05-28T16:37:49.960Z [DEBUG] nomad.vault: successfully renewed server token
2020-05-28T16:37:49.960Z [INFO]  nomad.vault: successfully renewed token: next_renewal=35h59m59.999915485s
2020-05-28T16:37:50.031Z [DEBUG] consul.sync: sync complete: registered_services=3 deregistered_services=0 registered_checks=3 deregistered_checks=0
2020-05-28T16:37:50.066Z [DEBUG] consul.sync: sync complete: registered_services=1 deregistered_services=0 registered_checks=1 deregistered_checks=0
2020-05-28T16:37:51.073Z [ERROR] http: request failed: method=GET path=/v1/agent/self error="ACL token not found" code=403
2020-05-28T16:37:51.073Z [DEBUG] http: request complete: method=GET path=/v1/agent/self duration=456.303µs
2020-05-28T16:37:51.098Z [WARN]  nomad.raft: heartbeat timeout reached, starting election: last-leader=
2020-05-28T16:37:51.098Z [INFO]  nomad.raft: entering candidate state: node="Node at 192.168.50.93:4647 [Candidate]" term=8
2020-05-28T16:37:51.107Z [DEBUG] nomad.raft: votes: needed=2
2020-05-28T16:37:51.107Z [DEBUG] nomad.raft: vote granted: from=192.168.50.92:4647 term=8 tally=1
2020-05-28T16:37:51.107Z [DEBUG] nomad.raft: vote granted: from=192.168.50.93:4647 term=8 tally=2
2020-05-28T16:37:51.107Z [INFO]  nomad.raft: election won: tally=2
2020-05-28T16:37:51.107Z [INFO]  nomad.raft: entering leader state: leader="Node at 192.168.50.93:4647 [Leader]"
2020-05-28T16:37:51.107Z [INFO]  nomad.raft: added peer, starting replication: peer=192.168.50.92:4647
2020-05-28T16:37:51.107Z [INFO]  nomad.raft: added peer, starting replication: peer=192.168.50.91:4647
2020-05-28T16:37:51.107Z [INFO]  nomad: cluster leadership acquired
2020-05-28T16:37:51.109Z [INFO]  nomad.raft: pipelining replication: peer="{Voter 192.168.50.92:4647 192.168.50.92:4647}"
2020-05-28T16:37:51.113Z [INFO]  nomad.raft: pipelining replication: peer="{Voter 192.168.50.91:4647 192.168.50.91:4647}"
2020-05-28T16:37:51.246Z [INFO]  client: node registration complete
2020-05-28T16:37:52.272Z [DEBUG] http: request complete: method=GET path=/v1/agent/self duration=545.109µs
2020-05-28T16:37:52.273Z [DEBUG] http: request complete: method=GET path=/v1/nodes?prefix=8cd0bef0-a6ac-a721-da26-cc9975bc2fc9 duration=410.058µs
2020-05-28T16:37:52.274Z [DEBUG] http: request complete: method=GET path=/v1/node/8cd0bef0-a6ac-a721-da26-cc9975bc2fc9 duration=132.628µs
2020-05-28T16:37:52.275Z [DEBUG] http: request complete: method=PUT path=/v1/node/8cd0bef0-a6ac-a721-da26-cc9975bc2fc9/eligibility duration=135.786µs
2020-05-28T16:37:56.789Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=245.648µs
2020-05-28T16:37:57.344Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:39820
2020-05-28T16:37:57.452Z [DEBUG] client: state updated: node_status=ready
2020-05-28T16:37:57.452Z [DEBUG] client.server_mgr: new server list: new_servers=[192.168.50.91:4647, 192.168.50.92:4647, 192.168.50.93:4647] old_servers=[0.0.0.0:4647, 192.168.50.93:4647]
2020-05-28T16:37:58.026Z [DEBUG] client: state changed, updating node and re-registering
2020-05-28T16:37:58.040Z [INFO]  client: node registration complete
2020-05-28T16:37:59.734Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=88.254µs
2020-05-28T16:38:06.793Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=247.484µs
2020-05-28T16:38:07.345Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:39832
2020-05-28T16:38:09.735Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=79.766µs
2020-05-28T16:38:13.795Z [DEBUG] http: request complete: method=GET path=/v1/agent/self duration=452.951µs
2020-05-28T16:38:13.796Z [DEBUG] http: request complete: method=GET path=/v1/nodes?prefix=8cd0bef0-a6ac-a721-da26-cc9975bc2fc9 duration=194.992µs
2020-05-28T16:38:13.797Z [DEBUG] http: request complete: method=GET path=/v1/node/8cd0bef0-a6ac-a721-da26-cc9975bc2fc9 duration=234.48µs
2020-05-28T16:38:13.798Z [DEBUG] http: request complete: method=PUT path=/v1/node/8cd0bef0-a6ac-a721-da26-cc9975bc2fc9/eligibility duration=125.366µs
2020-05-28T16:38:16.794Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=85.335µs
2020-05-28T16:38:17.346Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:39850
2020-05-28T16:38:19.737Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=170.11µs
2020-05-28T16:38:26.799Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=145.007µs
2020-05-28T16:38:27.347Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:39858
2020-05-28T16:38:29.739Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=76.45µs
2020-05-28T16:38:36.801Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=88.596µs
2020-05-28T16:38:37.350Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:39868
2020-05-28T16:38:39.740Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=78.544µs
2020-05-28T16:38:46.803Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=145.851µs
2020-05-28T16:38:47.351Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:39876
2020-05-28T16:38:49.750Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=7.39838ms
2020-05-28T16:38:56.818Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=111.901µs
2020-05-28T16:38:57.352Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:39884
2020-05-28T16:38:59.752Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=73.213µs
2020-05-28T16:39:06.820Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=166.814µs
2020-05-28T16:39:07.353Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:39894
2020-05-28T16:39:09.753Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=132.614µs
2020-05-28T16:39:16.822Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=97.483µs
2020-05-28T16:39:17.353Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:39904
2020-05-28T16:39:19.829Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=156.637µs
2020-05-28T16:39:25.710Z [DEBUG] nomad: memberlist: Stream connection from=192.168.50.91:37568
2020-05-28T16:39:26.824Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=124.702µs
2020-05-28T16:39:27.353Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:39912
2020-05-28T16:39:29.847Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=122.521µs
2020-05-28T16:39:36.826Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=650.676µs
2020-05-28T16:39:37.361Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:39922
2020-05-28T16:39:39.858Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=105.956µs
2020-05-28T16:39:46.827Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=134.209µs
2020-05-28T16:39:47.365Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:39930
2020-05-28T16:39:49.568Z [DEBUG] nomad: memberlist: Initiating push/pull sync with: 192.168.50.92:4648
2020-05-28T16:39:49.865Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=97.209µs
2020-05-28T16:39:56.829Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration=133.027µs
2020-05-28T16:39:57.365Z [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:39940
2020-05-28T16:39:59.877Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=156.489µs