Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Nomad deregisters third-party services from Consul #560

Closed
c4milo opened this issue Dec 10, 2015 · 4 comments
Closed

Nomad deregisters third-party services from Consul #560

c4milo opened this issue Dec 10, 2015 · 4 comments

Comments

@c4milo
Copy link
Contributor

c4milo commented Dec 10, 2015

I'm running Consul 0.6.0 with bootstrap 1 and one service definition under /etc/consul.d/services. Once Consul is started, I can see the service gets registered but it gets deregistered immediately thereafter by Nomad. I'm running no jobs in Nomad. It's a clean installation.

Consul invocation

/usr/bin/consul agent -config-dir=/etc/consul.d -config-dir=/etc/consul.d/services

Consul configuration

{
  "data_dir": "/var/lib/consul",
  "log_level": "DEBUG",
  "enable_syslog": true,
  "datacenter": "dc1",
  "retry_join": ["localhost"],
  "server": true,
  "bootstrap_expect": 1
}

Service definition

{
  "service": {
    "name": "nomad",
    "port": 4647
  }
}

Consul Log

Dec 10 16:09:13 dev systemd[1]: Started Consul Agent.
Dec 10 16:09:13 dev consul[23930]: ==> WARNING: LAN keyring exists but -encrypt given, using keyring
Dec 10 16:09:13 dev consul[23930]: ==> WARNING: WAN keyring exists but -encrypt given, using keyring
Dec 10 16:09:13 dev consul[23930]: ==> WARNING: BootstrapExpect Mode is specified as 1; this is the same as Bootstrap mode.
Dec 10 16:09:13 dev consul[23930]: ==> WARNING: Bootstrap mode enabled! Do not enable unless necessary
Dec 10 16:09:13 dev consul[23930]: ==> Starting Consul agent...
Dec 10 16:09:13 dev consul[23930]: raft: Node at 127.0.0.1:8300 [Follower] entering Follower state
Dec 10 16:09:13 dev consul[23930]: serf: EventMemberJoin: dev 127.0.0.1
Dec 10 16:09:13 dev consul[23930]: serf: EventMemberJoin: dev.dc1 127.0.0.1
Dec 10 16:09:13 dev consul[23930]: consul: adding LAN server dev (Addr: 127.0.0.1:8300) (DC: dc1)
Dec 10 16:09:13 dev consul[23930]: consul: adding WAN server dev.dc1 (Addr: 127.0.0.1:8300) (DC: dc1)
Dec 10 16:09:13 dev consul[23930]: ==> Starting Consul agent RPC...
Dec 10 16:09:13 dev consul[23930]: ==> Consul agent running!
Dec 10 16:09:13 dev consul[23930]: Node name: 'dev'
Dec 10 16:09:13 dev consul[23930]: Datacenter: 'dc1'
Dec 10 16:09:13 dev consul[23930]: Server: true (bootstrap: true)
Dec 10 16:09:13 dev consul[23930]: Client Addr: 127.0.0.1 (HTTP: 8500, HTTPS: -1, DNS: 8600, RPC: 8400)
Dec 10 16:09:13 dev consul[23930]: Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
Dec 10 16:09:13 dev consul[23930]: Gossip encrypt: true, RPC-TLS: false, TLS-Incoming: false
Dec 10 16:09:13 dev consul[23930]: Atlas: <disabled>
Dec 10 16:09:13 dev consul[23930]: ==> Log data will now stream in as it occurs:
Dec 10 16:09:13 dev consul[23930]: 2015/12/10 16:09:13 [INFO] raft: Node at 127.0.0.1:8300 [Follower] entering Follower state
Dec 10 16:09:13 dev consul[23930]: 2015/12/10 16:09:13 [INFO] serf: EventMemberJoin: dev 127.0.0.1
Dec 10 16:09:13 dev consul[23930]: 2015/12/10 16:09:13 [INFO] serf: EventMemberJoin: dev.dc1 127.0.0.1
Dec 10 16:09:13 dev consul[23930]: 2015/12/10 16:09:13 [INFO] consul: adding LAN server dev (Addr: 127.0.0.1:8300) (DC: dc1)
Dec 10 16:09:13 dev consul[23930]: 2015/12/10 16:09:13 [INFO] consul: adding WAN server dev.dc1 (Addr: 127.0.0.1:8300) (DC: dc1)
Dec 10 16:09:13 dev consul[23930]: agent: failed to sync remote state: No cluster leader
Dec 10 16:09:13 dev consul[23930]: agent: Joining cluster...
Dec 10 16:09:13 dev consul[23930]: agent: (LAN) joining: [localhost]
Dec 10 16:09:13 dev consul[23930]: memberlist: Initiating push/pull sync with: [::1]:8301
Dec 10 16:09:13 dev consul[23930]: memberlist: TCP connection from: [::1]:40919
Dec 10 16:09:13 dev consul[23930]: 2015/12/10 16:09:13 [ERR] agent: failed to sync remote state: No cluster leader
Dec 10 16:09:13 dev consul[23930]: 2015/12/10 16:09:13 [INFO] agent: Joining cluster...
Dec 10 16:09:13 dev consul[23930]: 2015/12/10 16:09:13 [INFO] agent: (LAN) joining: [localhost]
Dec 10 16:09:13 dev consul[23930]: 2015/12/10 16:09:13 [DEBUG] memberlist: Initiating push/pull sync with: [::1]:8301
Dec 10 16:09:13 dev consul[23930]: 2015/12/10 16:09:13 [DEBUG] memberlist: TCP connection from: [::1]:40919
Dec 10 16:09:13 dev consul[23930]: memberlist: Initiating push/pull sync with: 127.0.0.1:8301
Dec 10 16:09:13 dev consul[23930]: memberlist: TCP connection from: 127.0.0.1:37652
Dec 10 16:09:13 dev consul[23930]: agent: (LAN) joined: 2 Err: <nil>
Dec 10 16:09:13 dev consul[23930]: agent: Join completed. Synced with 2 initial agents
Dec 10 16:09:13 dev consul[23930]: 2015/12/10 16:09:13 [DEBUG] memberlist: Initiating push/pull sync with: 127.0.0.1:8301
Dec 10 16:09:13 dev consul[23930]: 2015/12/10 16:09:13 [DEBUG] memberlist: TCP connection from: 127.0.0.1:37652
Dec 10 16:09:13 dev consul[23930]: 2015/12/10 16:09:13 [INFO] agent: (LAN) joined: 2 Err: <nil>
Dec 10 16:09:13 dev consul[23930]: 2015/12/10 16:09:13 [INFO] agent: Join completed. Synced with 2 initial agents
Dec 10 16:09:13 dev consul[23930]: 2015/12/10 16:09:13 [DEBUG] http: Request GET /v1/agent/self (524.421µs)
Dec 10 16:09:13 dev consul[23930]: http: Request GET /v1/agent/self (524.421µs)
Dec 10 16:09:15 dev consul[23930]: 2015/12/10 16:09:15 [WARN] raft: Heartbeat timeout reached, starting election
Dec 10 16:09:15 dev consul[23930]: raft: Heartbeat timeout reached, starting election
Dec 10 16:09:15 dev consul[23930]: raft: Node at 127.0.0.1:8300 [Candidate] entering Candidate state
Dec 10 16:09:15 dev consul[23930]: 2015/12/10 16:09:15 [INFO] raft: Node at 127.0.0.1:8300 [Candidate] entering Candidate state
Dec 10 16:09:15 dev consul[23930]: 2015/12/10 16:09:15 [DEBUG] raft: Votes needed: 1
Dec 10 16:09:15 dev consul[23930]: 2015/12/10 16:09:15 [DEBUG] raft: Vote granted. Tally: 1
Dec 10 16:09:15 dev consul[23930]: 2015/12/10 16:09:15 [INFO] raft: Election won. Tally: 1
Dec 10 16:09:15 dev consul[23930]: raft: Votes needed: 1
Dec 10 16:09:15 dev consul[23930]: raft: Vote granted. Tally: 1
Dec 10 16:09:15 dev consul[23930]: raft: Election won. Tally: 1
Dec 10 16:09:15 dev consul[23930]: raft: Node at 127.0.0.1:8300 [Leader] entering Leader state
Dec 10 16:09:15 dev consul[23930]: consul: cluster leadership acquired
Dec 10 16:09:15 dev consul[23930]: consul: New leader elected: dev
Dec 10 16:09:15 dev consul[23930]: raft: Disabling EnableSingleNode (bootstrap)
Dec 10 16:09:15 dev consul[23930]: raft: Node 127.0.0.1:8300 updated peer set (2): [127.0.0.1:8300]
Dec 10 16:09:15 dev consul[23930]: raft: Node 127.0.0.1:8300 updated peer set (2): [127.0.0.1:8300]
Dec 10 16:09:15 dev consul[23930]: 2015/12/10 16:09:15 [INFO] raft: Node at 127.0.0.1:8300 [Leader] entering Leader state
Dec 10 16:09:15 dev consul[23930]: 2015/12/10 16:09:15 [INFO] consul: cluster leadership acquired
Dec 10 16:09:15 dev consul[23930]: 2015/12/10 16:09:15 [INFO] consul: New leader elected: dev
Dec 10 16:09:15 dev consul[23930]: 2015/12/10 16:09:15 [INFO] raft: Disabling EnableSingleNode (bootstrap)
Dec 10 16:09:15 dev consul[23930]: 2015/12/10 16:09:15 [DEBUG] raft: Node 127.0.0.1:8300 updated peer set (2): [127.0.0.1:8300]
Dec 10 16:09:15 dev consul[23930]: 2015/12/10 16:09:15 [DEBUG] raft: Node 127.0.0.1:8300 updated peer set (2): [127.0.0.1:8300]
Dec 10 16:09:15 dev consul[23930]: 2015/12/10 16:09:15 [DEBUG] raft: Node 127.0.0.1:8300 updated peer set (2): [127.0.0.1:8300]
Dec 10 16:09:15 dev consul[23930]: raft: Node 127.0.0.1:8300 updated peer set (2): [127.0.0.1:8300]
Dec 10 16:09:15 dev consul[23930]: raft: Node 127.0.0.1:8300 updated peer set (2): [127.0.0.1:8300]
Dec 10 16:09:15 dev consul[23930]: raft: Node 127.0.0.1:8300 updated peer set (2): [127.0.0.1:8300]
Dec 10 16:09:15 dev consul[23930]: raft: Node 127.0.0.1:8300 updated peer set (2): [127.0.0.1:8300]
Dec 10 16:09:15 dev consul[23930]: raft: Node 127.0.0.1:8300 updated peer set (2): [127.0.0.1:8300]
Dec 10 16:09:15 dev consul[23930]: 2015/12/10 16:09:15 [DEBUG] raft: Node 127.0.0.1:8300 updated peer set (2): [127.0.0.1:8300]
Dec 10 16:09:15 dev consul[23930]: 2015/12/10 16:09:15 [DEBUG] raft: Node 127.0.0.1:8300 updated peer set (2): [127.0.0.1:8300]
Dec 10 16:09:15 dev consul[23930]: 2015/12/10 16:09:15 [DEBUG] raft: Node 127.0.0.1:8300 updated peer set (2): [127.0.0.1:8300]
Dec 10 16:09:15 dev consul[23930]: 2015/12/10 16:09:15 [DEBUG] raft: Node 127.0.0.1:8300 updated peer set (2): [127.0.0.1:8300]
Dec 10 16:09:15 dev consul[23930]: 2015/12/10 16:09:15 [DEBUG] consul: reset tombstone GC to index 63
Dec 10 16:09:15 dev consul[23930]: consul: reset tombstone GC to index 63
Dec 10 16:09:17 dev consul[23930]: 2015/12/10 16:09:17 [DEBUG] agent: Service 'consul' in sync
Dec 10 16:09:17 dev consul[23930]: agent: Service 'consul' in sync
Dec 10 16:09:17 dev consul[23930]: 2015/12/10 16:09:17 [INFO] agent: Synced service 'nomad'
Dec 10 16:09:17 dev consul[23930]: agent: Synced service 'nomad'
Dec 10 16:09:18 dev consul[23930]: 2015/12/10 16:09:18 [DEBUG] http: Request GET /v1/agent/services (284.572µs)
Dec 10 16:09:18 dev consul[23930]: http: Request GET /v1/agent/services (284.572µs)
Dec 10 16:09:18 dev consul[23930]: 2015/12/10 16:09:18 [DEBUG] http: Request GET /v1/agent/checks (181.834µs)
Dec 10 16:09:18 dev consul[23930]: http: Request GET /v1/agent/checks (181.834µs)
Dec 10 16:09:18 dev consul[23930]: agent: Service 'consul' in sync
Dec 10 16:09:18 dev consul[23930]: 2015/12/10 16:09:18 [DEBUG] agent: Service 'consul' in sync
Dec 10 16:09:18 dev consul[23930]: 2015/12/10 16:09:18 [INFO] agent: Deregistered service 'nomad'
Dec 10 16:09:18 dev consul[23930]: 2015/12/10 16:09:18 [DEBUG] http: Request PUT /v1/agent/service/deregister/nomad (1.715655ms)
Dec 10 16:09:18 dev consul[23930]: agent: Deregistered service 'nomad'
Dec 10 16:09:18 dev consul[23930]: http: Request PUT /v1/agent/service/deregister/nomad (1.715655ms)
Dec 10 16:09:18 dev consul[23930]: agent: Service 'consul' in sync
Dec 10 16:09:18 dev consul[23930]: 2015/12/10 16:09:18 [DEBUG] agent: Service 'consul' in sync
Dec 10 16:09:23 dev consul[23930]: 2015/12/10 16:09:23 [DEBUG] http: Request GET /v1/agent/services (36.475µs)
Dec 10 16:09:23 dev consul[23930]: http: Request GET /v1/agent/services (36.475µs)
Dec 10 16:09:23 dev consul[23930]: http: Request GET /v1/agent/checks (13.511µs)
Dec 10 16:09:23 dev consul[23930]: 2015/12/10 16:09:23 [DEBUG] http: Request GET /v1/agent/checks (13.511µs)
Dec 10 16:09:28 dev consul[23930]: 2015/12/10 16:09:28 [DEBUG] http: Request GET /v1/agent/services (34.487µs)
Dec 10 16:09:28 dev consul[23930]: http: Request GET /v1/agent/services (34.487µs)
Dec 10 16:09:28 dev consul[23930]: http: Request GET /v1/agent/checks (10.691µs)
Dec 10 16:09:28 dev consul[23930]: 2015/12/10 16:09:28 [DEBUG] http: Request GET /v1/agent/checks (10.691µs)
Dec 10 16:09:28 dev consul[23930]: 2015/12/10 16:09:28 [DEBUG] http: Request GET /v1/agent/self (216.206µs)
Dec 10 16:09:28 dev consul[23930]: http: Request GET /v1/agent/self (216.206µs)
Dec 10 16:09:33 dev consul[23930]: 2015/12/10 16:09:33 [DEBUG] http: Request GET /v1/agent/services (36.074µs)
Dec 10 16:09:33 dev consul[23930]: 2015/12/10 16:09:33 [DEBUG] http: Request GET /v1/agent/checks (10.205µs)
Dec 10 16:09:33 dev consul[23930]: http: Request GET /v1/agent/services (36.074µs)
Dec 10 16:09:33 dev consul[23930]: http: Request GET /v1/agent/checks (10.205µs)
Dec 10 16:09:38 dev consul[23930]: 2015/12/10 16:09:38 [DEBUG] http: Request GET /v1/agent/services (34.381µs)
Dec 10 16:09:38 dev consul[23930]: http: Request GET /v1/agent/services (34.381µs)
Dec 10 16:09:38 dev consul[23930]: http: Request GET /v1/agent/checks (10.226µs)
Dec 10 16:09:38 dev consul[23930]: 2015/12/10 16:09:38 [DEBUG] http: Request GET /v1/agent/checks (10.226µs)
Dec 10 16:09:43 dev consul[23930]: 2015/12/10 16:09:43 [DEBUG] http: Request GET /v1/agent/services (34.364µs)
Dec 10 16:09:43 dev consul[23930]: http: Request GET /v1/agent/services (34.364µs)
Dec 10 16:09:43 dev consul[23930]: http: Request GET /v1/agent/checks (12.827µs)

Nomad invocation

/usr/bin/nomad agent -dev -config=/etc/nomad.d

Nomad configuration

data_dir = "/var/lib/nomad"

# Exit gracefully
leave_on_interrupt = true

log_level = "INFO"
enable_syslog = true
syslog_facility = "LOCAL0"
datacenter = "dc1"
@c4milo c4milo mentioned this issue Dec 10, 2015
Closed
@dadgar
Copy link
Contributor

dadgar commented Dec 11, 2015

Hey, this is a known bug. A fix will be released soon.

@c4milo c4milo mentioned this issue Dec 11, 2015
Closed
@diptanu diptanu mentioned this issue Dec 11, 2015
Merged
@diptanu diptanu closed this as completed Dec 11, 2015
@diptanu
Copy link
Contributor

diptanu commented Dec 11, 2015

@c4milo The PR has been merged

@c4milo
Copy link
Contributor Author

c4milo commented Dec 11, 2015

Awesome! Thank you!
On Fri, Dec 11, 2015 at 5:30 PM Diptanu Choudhury [email protected]
wrote:

@c4milo https://github.com/c4milo The PR has been merged

—
Reply to this email directly or view it on GitHub
#560 (comment).

@github-actions
Copy link

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Dec 28, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@diptanu @c4milo @dadgar