Add sensu_cluster_member type #947

treydock · 2018-08-05T17:20:11Z

Pull Request Checklist

Description

Add a custom type that can add cluster members.

Related Issue

Part of #901

Motivation and Context

sensuctl supports adding members. The members won't actually be fully added until the new member is started with the proper configuration. The necessary configuration is printed by this custom type in the info output which comes from output of the sensuctl command to add the member.

How Has This Been Tested?

Added new way to run acceptance tests that spawn 3 backend servers. Using only 2 caused issues (see sensu/sensu-go#1890). When BEAKER_sensu_cluster is yes or true the cluster acceptance tests are run.

General

Update README.md with any necessary configuration snippets
Tests pass - bundle exec rake validate lint spec

ghoneycutt · 2018-08-05T23:19:28Z

Vagrantfile

+  config.vm.define "sensu-backend-peer", primary: true, autostart: true do |backend|
+    backend.vm.box = "centos/7"
+    backend.vm.hostname = 'sensu-backend-peer.example.com'
+    backend.vm.network :private_network, ip: ENV['ALTERNATE_IP'] || '192.168.52.11'


these IP's conflict with other vagrant instances

ghoneycutt · 2018-08-05T23:20:11Z

Vagrantfile

+    backend.vm.network :forwarded_port, guest: 2380, host: 2380, auto_correct: true
+    backend.vm.network :forwarded_port, guest: 3000, host: 3000, auto_correct: true
+    backend.vm.network :forwarded_port, guest: 8080, host: 8080, auto_correct: true
+    backend.vm.network :forwarded_port, guest: 8081, host: 8081, auto_correct: true


These ports conflict. I know auto_correct is true, though that makes documentation hard since a random port will be grabbed.

ghoneycutt · 2018-08-05T23:22:23Z

lib/puppet/provider/sensu_cluster_member/sensuctl.rb

@@ -0,0 +1,98 @@
+require File.expand_path(File.join(File.dirname(__FILE__), '..', 'sensuctl'))


The README should have an example for how this works.

There is an example in the type description that will get picked up with puppet-strings, do we also want additional examples in README? If yes, should we also document how to setup a cluster? It's currently not documented by Sensu but managed to figure it out based on their docker compose configs. This new type is tightly coupled to configs in backend.yml as they won't work without both being implemented.

Let's document how to setup a cluster since that's how people will run Sensu.

ghoneycutt · 2018-08-08T00:41:56Z

@treydock please squash and rebase and this should pass tests once #948 is merged.

treydock · 2018-08-09T22:46:26Z

@ghoneycutt Squashed and rebased. Updated README with instructions on how to setup a cluster and also how to add a member to an existing cluster.

ghoneycutt · 2018-08-09T23:14:29Z

Vagrantfile

@@ -46,6 +46,32 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
    backend.vm.provision :shell, :inline => "facter --custom-dir=/vagrant/lib/facter sensu_version"
  end

+  config.vm.define "sensu-backend-peer", primary: true, autostart: true do |backend|


call this one sensu-backend-peer1 to be inline with sensu-backend-peer2

ghoneycutt · 2018-08-09T23:15:25Z

README.md

+
+Adding new members to a cluster requires two steps. First add the member using `sensu_cluster_member` type then second configure and start `sensu-backend` on the new node.
+
+First, add the member:


Does this mean it takes multiple puppet runs in a certain sequence? If so, could you elaborate?

I think so, I'll try to come up with some better wording on order. It's a really specific order of operations to avoid errors. Basically add member on existing backend node with sensuctl then start new member with config that differs slightly from existing backend nodes.

Updated README, hopefully a bit clearer. Let me know if needs improvements.

ghoneycutt · 2018-08-09T23:16:30Z

This strangely had an error with amazon linux 2018.3. Perhaps the service just took too long to start?

ghoneycutt · 2018-08-10T01:11:15Z

The docs assume that two backends are running, which I thought is not good since you need one or three for quorum. If you have only two does is put the cluster is a not-good state? Could you give me a step by step in the README of why you have to setup one node, then can add more.

treydock · 2018-08-10T12:21:33Z

So the adding member bits are only necessary if your going from an existing cluster of say 2 nodes and want to add a third node later on. The easiest way I found to setup a cluster is to have all nodes from the very beginning, which doesn't require the sensu_cluster_member type at all, just YAML configs. The sensu_cluster_member type would only be for modifying an already established cluster. Going from a 1 node cluster to a 2 node cluster presents problems which I've raised on sensu-go (sensu/sensu-go#1890) as seems the cluster looses quorum and breaks sensuctl until the 2nd node is brought online.

treydock · 2018-08-11T17:08:41Z

Added a few bits of text to hopefully make it clear that adding members to an existing cluster differs from setting up a cluster from scratch.

ghoneycutt · 2018-08-25T20:01:05Z

@treydock Could you please rebase. Is this ready to merge?

Update README with backend cluster examples

treydock · 2018-08-26T17:09:03Z

@ghoneycutt rebased and squashed.

treydock · 2018-08-28T23:57:25Z

I'd say ready for merge. Open issue upstream but not sure would impact this PR since issue I opened was going from one to two node cluster and sensuctl hanging. That's why acceptance tests go from two to three node cluster, seems keeps quorum.

treydock force-pushed the sensu2-cluster-member branch 2 times, most recently from 6454ddf to 6578dd0 Compare August 5, 2018 17:46

ghoneycutt requested changes Aug 5, 2018

View reviewed changes

treydock mentioned this pull request Aug 8, 2018

Fix acceptance tests #948

Merged

treydock force-pushed the sensu2-cluster-member branch from c45ae4f to 9b990e8 Compare August 9, 2018 22:45

ghoneycutt reviewed Aug 9, 2018

View reviewed changes

treydock force-pushed the sensu2-cluster-member branch 3 times, most recently from e5aeb8e to d9e93f0 Compare August 10, 2018 00:29

treydock force-pushed the sensu2-cluster-member branch from c3867b3 to 621f442 Compare August 21, 2018 23:15

ghoneycutt added the sensu v2 label Aug 25, 2018

Add sensu_cluster_member type

7879b1c

Update README with backend cluster examples

treydock force-pushed the sensu2-cluster-member branch from 621f442 to 7879b1c Compare August 26, 2018 17:08

ghoneycutt approved these changes Aug 29, 2018

View reviewed changes

ghoneycutt merged commit b4a2b5b into sensu:sensu2 Aug 29, 2018

treydock deleted the sensu2-cluster-member branch August 31, 2018 00:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add sensu_cluster_member type #947

Add sensu_cluster_member type #947

treydock commented Aug 5, 2018 •

edited

Loading

ghoneycutt Aug 5, 2018

ghoneycutt Aug 5, 2018

ghoneycutt Aug 5, 2018

treydock Aug 7, 2018

ghoneycutt Aug 8, 2018

ghoneycutt commented Aug 8, 2018

treydock commented Aug 9, 2018

ghoneycutt Aug 9, 2018

ghoneycutt Aug 9, 2018

treydock Aug 9, 2018

treydock Aug 10, 2018

ghoneycutt commented Aug 9, 2018

ghoneycutt commented Aug 10, 2018

treydock commented Aug 10, 2018

treydock commented Aug 11, 2018

ghoneycutt commented Aug 25, 2018

treydock commented Aug 26, 2018

treydock commented Aug 28, 2018

		@@ -0,0 +1,98 @@
		require File.expand_path(File.join(File.dirname(__FILE__), '..', 'sensuctl'))


		Adding new members to a cluster requires two steps. First add the member using `sensu_cluster_member` type then second configure and start `sensu-backend` on the new node.

		First, add the member:

Add sensu_cluster_member type #947

Add sensu_cluster_member type #947

Conversation

treydock commented Aug 5, 2018 • edited Loading

Pull Request Checklist

Description

Related Issue

Motivation and Context

How Has This Been Tested?

General

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ghoneycutt commented Aug 8, 2018

treydock commented Aug 9, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ghoneycutt commented Aug 9, 2018

ghoneycutt commented Aug 10, 2018

treydock commented Aug 10, 2018

treydock commented Aug 11, 2018

ghoneycutt commented Aug 25, 2018

treydock commented Aug 26, 2018

treydock commented Aug 28, 2018

treydock commented Aug 5, 2018 •

edited

Loading