fixes #3. Config for external access to k8s

Default is incluster but handy to have
external access for testing.

CONTRIBUTING.md

@@ -0,0 +1,25 @@
+# Contributing guidelines
+
+## Filing issues
+
+File issues using the standard Github issue tracker for the repo.
+
+# Building the code
+
+1. * [golang 1.5+](https://golang.org/dl/)
+2. Run ./build.sh
+3. To run the tests: `go test -v ./kubestate`
+
+You will need a Kubernetes cluster to fetch metrics from. A quick way to get going is to use [Minikube](https://github.com/kubernetes/minikube).
+
+You can use client_test.go as way to test if you can successfully contact the cluster. Remove `Skip` from `SkipConvey` to enable the tests and then run `go test -v ./kubestate`
+
+To test the snap task locally with the snap server (snapd or snapteld):
+
+1. Load the plugin:
+  `snapctl plugin load build/snap-plugin-collector-kubestate` or `snaptel plugin load build/snap-plugin-collector-kubestate` if you have latest version of Snap.
+2. If you want to publish the results to a file, you need to download the [file publisher plugin](https://s3-us-west-2.amazonaws.com/snap.ci.snap-telemetry.io/plugins/snap-plugin-publisher-file/latest/linux/x86_64/snap-plugin-publisher-file) and load that too:
+  `snapctl plugin load snap-plugin-publisher-file_linux_x86_64`
+2. Create a task using the [example task](examples/task.json) after editing the config to point to your Kubernetes config file:
+  `snapctl task create -t examples/task.json`
+3. `snapctl task list` to check if it is running and then `snapctl task watch <task id>` to see the value being fetched.

README.md

@@ -1,4 +1,5 @@
 # snap collector plugin - kube state
+
 This plugin collects metrics from Kubernetes about the state of pods, nodes and deployments.
 
 It's used in the [snap framework](http://github.com:intelsdi-x/snap).
@@ -11,25 +12,36 @@ It's used in the [snap framework](http://github.com:intelsdi-x/snap).
   * [Collected Metrics](#collected-metrics)
   * [Examples](#examples)
   * [Roadmap](#roadmap)
-3. [Community Support](#community-support)
-4. [Contributing](#contributing)
-5. [License](#license-and-authors)
-6. [Acknowledgements](#acknowledgements)
+3. [Contributing](#contributing)
+4. [License](#license-and-authors)
+5. [Acknowledgements](#acknowledgements)
 
 ## Getting Started
+
 ### System Requirements
+
 * [golang 1.5+](https://golang.org/dl/) (needed only for building)
 
 ### Operating systems
+
 All OSs currently supported by snap:
 * Linux/amd64
 * Darwin/amd64
 
 ### Installation
+
+This plugin monitors Kubernetes so you need a Kubernetes cluster to monitor. A quick way to get a local test cluster installed is to use [Minikube](https://github.com/kubernetes/minikube).
+
+This plugin is included in the [Raintank Docker image for Kubernetes](https://github.com/raintank/snap_k8s) if you want to see an example of how to easily deploy it as pod to Kubernetes.
+
+This plugin and the above Docker image are used to collect metrics for the [Grafana Kubernetes App](https://github.com/raintank/kubernetes-app).
+
 #### Download kubestate plugin binary:
-You can get the pre-built binaries for your OS and architecture at snap's [GitHub Releases](https://github.com/raintank/snap-plugin-collector-kubestate/releases) page.
+
+You can get the pre-built binaries at [GitHub Releases](https://github.com/raintank/snap-plugin-collector-kubestate/releases) page.
 
 #### To build the plugin binary:
+
 Fork https://github.com/raintank/snap-plugin-collector-kubestate
 Clone repo into `$GOPATH/src/github.com/raintank/`:
 
@@ -39,21 +51,49 @@ $ git clone https://github.com/<yourGithubID>/snap-plugin-collector-kubestate.gi
 
 Build the plugin by running make within the cloned repo:
 ```
-$ make
+$ ./build.sh
 ```
-This builds the plugin in `/build/rootfs/`
+This builds the plugin binary in `/build/`
+
+This plugin uses govendor to manage dependencies. If you want to add a dependency, then:
+
+1. Install govendor with: `go get -u github.com/kardianos/govendor`
+2. `govendor fetch <dependency path>` e.g. `govendor fetch k8s.io/client-go/tools/clientcmd/[email protected]` The `...` means install all sub dependencies.
+3. `govendor install` to update the vendor.json file.
+4. Check in the new dependency that will be in the vendor directory.
 
 ### Configuration and Usage
+
 * Set up the [snap framework](https://github.com/intelsdi-x/snap/blob/master/README.md#getting-started)
 * Ensure `$SNAP_PATH` is exported
 `export SNAP_PATH=$GOPATH/src/github.com/intelsdi-x/snap/build`
+* If running the task outside of a kubernetes cluster rather than in a pod, then the following two config variables must be set:
+  - `incluster` expects a boolean, default is true.
+  - `kubeconfigpath` expects a string path to the Kubernetes config file, default is empty string.
+
+  Example of how to configure it in a json task manifest:
+  ```json
+  "workflow": {
+    "collect": {
+      "metrics": {
+        "/grafanalabs/kubestate/*":{}
+      },
+      "config": {
+        "/grafanalabs/kubestate": {
+          "incluster": false,
+          "kubeconfigpath": "/home/user/.kube/config"
+        }
+      },
+  ```
 
 ## Documentation
-There are a number of other resources you can review to learn to use this plugin:
 
+There are a number of other resources you can review to learn to use this plugin:
 
+- [The Kubernetes API spec](http://kubernetes.io/docs/api-reference/v1/definitions/). All the metrics are fetched via the API.
 
 ### Collected Metrics
+
 This plugin has the ability to gather the following metrics:
 
 #### Pods
@@ -93,17 +133,14 @@ Namespace | Description (optional)
 /grafanalabs/kubestate/deployment/[NAMESPACE]/[DEPLOYMENT]/status/observedgeneration | The generation sequence number after deployment.
 /grafanalabs/kubestate/deployment/[NAMESPACE]/[DEPLOYMENT]/status/targetedreplicas | Total number of non-terminated pods targeted by this deployment (their labels match the selector).
 /grafanalabs/kubestate/deployment/[NAMESPACE]/[DEPLOYMENT]/status/availablereplicas | Total number of available pods (ready for at least minReadySeconds) targeted by this deployment.
+/grafanalabs/kubestate/deployment/[NAMESPACE]/[DEPLOYMENT]/status/unavailablereplicas | Total number of unavailable pods targeted by this deployment.
 /grafanalabs/kubestate/deployment/[NAMESPACE]/[DEPLOYMENT]/spec/desiredreplicas | Number of desired pods.
 
 ### Examples
 
 ### Roadmap
 
-1. Deployment metrics
-
-## Community Support
-
-This repository is one of **many** plugins in **snap**, a powerful telemetry framework. See the full project at http://github.com/intelsdi-x/snap To reach out to other users, head to the [main framework](https://github.com/intelsdi-x/snap#community-support)
+1. Disk capacity for the cluster
 
 ## Contributing
 

kubestate/client.go

@@ -15,7 +15,7 @@ type Client struct {
 	clientset *kubernetes.Clientset
 }
 
-func NewClient(incluster bool, kubeconfigpath string) (*Client, error) {
+var newClient = func(incluster bool, kubeconfigpath string) (*Client, error) {
 	var config *rest.Config
 	var err error
 

kubestate/client_test.go

@@ -8,7 +8,7 @@ import (
 
 func TestClient(t *testing.T) {
 	SkipConvey("When creating client", t, func() {
-		c, err := NewClient(false, "/home/<user>/.kube/config")
+		c, err := newClient(false, "/home/<user>/.kube/config")
 
 		So(err, ShouldBeNil)
 		So(c, ShouldNotBeNil)

kubestate/kubestate.go

@@ -15,51 +15,73 @@ type Kubestate struct {
 // CollectMetrics collects metrics for testing
 func (n *Kubestate) CollectMetrics(mts []plugin.Metric) ([]plugin.Metric, error) {
 	LogDebug("request to collect metrics", "metric_count", len(mts))
-	metrics := make([]plugin.Metric, 0)
-
-	incluster := true
-	kubeconfigpath := ""
 
-	// incluster, err := mts[0].Config.GetBool("incluster")
-	// if err != nil {
-	// 	LogError("failed to fetch config value incluster.", "error", err)
-	// 	incluster = true
-	// }
+	incluster, err := mts[0].Config.GetBool("incluster")
+	if err != nil {
+		LogError("failed to fetch config value incluster.", "error", err)
+		incluster = true
+	}
 
-	// kubeconfigpath, err := mts[0].Config.GetString("kubeconfigpath")
-	// if err != nil {
-	// 	LogError("failed to fetch config value kubeconfigpath.", "error", err)
-	// 	return nil, err
-	// }
+	kubeconfigpath := ""
+	if !incluster {
+		kubeconfigpath, err = mts[0].Config.GetString("kubeconfigpath")
+		if err != nil {
+			LogError("failed to fetch config value kubeconfigpath.", "error", err)
+			return nil, err
+		}
+	}
 
-	client, err := NewClient(incluster, kubeconfigpath)
+	client, err := newClient(incluster, kubeconfigpath)
 	if err != nil {
 		LogError("failed to create Kubernetes api client.", "error", err)
 		return nil, err
 	}
 
+	metrics, err := collect(client, mts)
+	if err != nil {
+		return nil, err
+	}
+
+	LogDebug("collecting metrics completed", "metric_count", len(metrics))
+	return metrics, nil
+}
+
+var collect = func(client *Client, mts []plugin.Metric) ([]plugin.Metric, error) {
+	metrics := make([]plugin.Metric, 0)
+
 	pods, err := client.GetPods()
+	if err != nil {
+		return nil, err
+	}
+
 	podCollector := new(podCollector)
 	for _, p := range pods.Items {
 		podMetrics, _ := podCollector.Collect(mts, p)
 		metrics = append(metrics, podMetrics...)
 	}
 
 	nodes, err := client.GetNodes()
+	if err != nil {
+		return nil, err
+	}
+
 	nodeCollector := new(nodeCollector)
 	for _, n := range nodes.Items {
 		nodeMetrics, _ := nodeCollector.Collect(mts, n)
 		metrics = append(metrics, nodeMetrics...)
 	}
 
 	deployments, err := client.GetDeployments()
+	if err != nil {
+		return nil, err
+	}
+
 	deploymentCollector := new(deploymentCollector)
 	for _, d := range deployments.Items {
 		deploymentMetrics, _ := deploymentCollector.Collect(mts, d)
 		metrics = append(metrics, deploymentMetrics...)
 	}
 
-	LogDebug("collecting metrics completed", "metric_count", len(metrics))
 	return metrics, nil
 }
 
@@ -357,7 +379,7 @@ func (n *Kubestate) GetMetricTypes(cfg plugin.Config) ([]plugin.Metric, error) {
 
 func (f *Kubestate) GetConfigPolicy() (plugin.ConfigPolicy, error) {
 	policy := plugin.NewConfigPolicy()
-	policy.AddNewBoolRule([]string{"grafanalabs", "kubestate"}, "incluster", false)
+	policy.AddNewBoolRule([]string{"grafanalabs", "kubestate"}, "incluster", false, plugin.SetDefaultBool(true))
 	policy.AddNewStringRule([]string{"grafanalabs", "kubestate"}, "kubeconfigpath", false)
 	return *policy, nil
 }

kubestate/kubestate_test.go

@@ -0,0 +1,74 @@
+package kubestate
+
+import (
+	"testing"
+
+	"github.com/intelsdi-x/snap-plugin-lib-go/v1/plugin"
+	. "github.com/smartystreets/goconvey/convey"
+)
+
+func TestKubestate(t *testing.T) {
+	Convey("When collect metrics is called", t, func() {
+		var inclusterSpy bool
+		var kubeConfigSpy string
+
+		newClient = func(incluster bool, kubeconfigpath string) (*Client, error) {
+			c := &Client{}
+			inclusterSpy = incluster
+			kubeConfigSpy = kubeconfigpath
+			return c, nil
+		}
+
+		collect = func(client *Client, mts []plugin.Metric) ([]plugin.Metric, error) {
+			return make([]plugin.Metric, 0), nil
+		}
+
+		ks := new(Kubestate)
+
+		metrics, err := ks.CollectMetrics(metricWithInclusterConfig)
+		So(err, ShouldBeNil)
+		So(metrics, ShouldNotBeNil)
+		So(inclusterSpy, ShouldBeTrue)
+		So(kubeConfigSpy, ShouldEqual, "")
+
+		metrics, err = ks.CollectMetrics(metricWithKubeConfig)
+		So(err, ShouldBeNil)
+		So(metrics, ShouldNotBeNil)
+		So(inclusterSpy, ShouldBeFalse)
+		So(kubeConfigSpy, ShouldEqual, "/home/user/.kube/config")
+
+		metrics, err = ks.CollectMetrics(metricWithNoConfig)
+		So(err, ShouldBeNil)
+		So(metrics, ShouldNotBeNil)
+		So(inclusterSpy, ShouldBeTrue)
+		So(kubeConfigSpy, ShouldEqual, "")
+	})
+}
+
+var metricWithInclusterConfig = []plugin.Metric{
+	{
+		Namespace: plugin.NewNamespace("grafanalabs", "kubestate"),
+		Version:   1,
+		Config: plugin.Config{
+			"incluster": true,
+		},
+	},
+}
+
+var metricWithKubeConfig = []plugin.Metric{
+	{
+		Namespace: plugin.NewNamespace("grafanalabs", "kubestate"),
+		Version:   1,
+		Config: plugin.Config{
+			"incluster":      false,
+			"kubeconfigpath": "/home/user/.kube/config",
+		},
+	},
+}
+
+var metricWithNoConfig = []plugin.Metric{
+	{
+		Namespace: plugin.NewNamespace("grafanalabs", "kubestate"),
+		Version:   1,
+	},
+}