Add couchdb module #9406

berfinsari · 2018-12-05T20:18:51Z

No description provided.

elasticmachine · 2018-12-05T20:18:52Z

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

metricbeat/module/couchdb/server/server.go

metricbeat/module/couchdb/server/data.go

elasticmachine · 2018-12-06T07:36:10Z

Pinging @elastic/infrastructure

berfinsari · 2018-12-11T22:39:26Z

Is there anything that I can do for this unsuccessful check?

ruflin · 2018-12-12T14:06:34Z

@berfinsari Try to run make update and make fmt and then push again. This will generate / update a few files.

berfinsari · 2018-12-13T00:07:37Z

I ran make update and make fmt commands and nothing changes. Is there anything else I can do?

jsoriano · 2018-12-13T08:15:13Z

@berfinsari you need to run make update fmt from the beats directory so it is applied to all beats. In this case the outdated file is in x-pack/metricbeat/, you can also run make update fmt from this directory.

jsoriano

This is looking good, I have added some comments, my main concern is that I am not sure if we should collect all aggregated values or only current values.

metricbeat/module/couchdb/server/_meta/fields.yml

metricbeat/module/couchdb/server/data.go

metricbeat/module/couchdb/_meta/test/serverstats.json

jsoriano · 2018-12-14T18:35:21Z

@berfinsari btw, I have seen you are also the author of other modules and some community beats for the same services, out of curiosity, do you have some preferences about using specific community beats instead of Metricbeat?

metricbeat/module/couchdb/_meta/test/serverstats.json

ruflin · 2018-12-17T10:50:50Z

metricbeat/module/couchdb/server/data.go

+		"httpd_status_codes": common.MapStr{
+			"200": common.MapStr{
+				"description": data.HttpdStatusCodes.Num200.Description,
+				"current":     data.HttpdStatusCodes.Num200.Current,


What exactly means current in this context? Since the last request?

I'm not sure that is the answer of your question, but the fields provide the current, minimum and maximum, and a collection of statistical means and quantities. These statistics produce results that measure the interaction with the server since it was started.

berfinsari · 2018-12-19T23:15:43Z

@jsoriano At the beginning of the process, I was creating community beats for both learning the process and preparing and using it, because there is not so many requirements like the metricbeat module in community beats. When I was keeping contribute to Metricbeat module, I was also keeping create community beats. I guess it is enough to contribute to Metricbeat module.

jsoriano · 2018-12-20T09:42:07Z

@jsoriano At the beginning of the process, I was creating community beats for both learning the process and preparing and using it, because there is not so many requirements like the metricbeat module in community beats. When I was keeping contribute to Metricbeat module, I was also keeping create community beats. I guess it is enough to contribute to Metricbeat module.

Sounds good, thanks a lot for your work in these modules!

sayden · 2018-12-20T09:54:56Z

Please, also write in the docs that this module works with Couchdb and it's not tested with Couchdb 2. While they both maintain a 99% of API compatibility, the amount of metrics you can extract from Couchdb 2 is bigger than in Couchdb 1, for example the shards of the databases or the statistics about Mango queries.

jsoriano · 2018-12-20T10:23:30Z

metricbeat/module/couchdb/server/_meta/fields.yml

+      description: >
+        Example field
+      fields:
+        - name: num200


I think fields here should be named just with the number i.e. 200 instead of num200.

metricbeat/module/couchdb/_meta/fields.yml

metricbeat/module/couchdb/server/_meta/fields.yml

jsoriano · 2019-01-09T13:23:36Z

metricbeat/module/couchdb/server/server.go

+
+// New creates a new instance of the MetricSet.
+func New(base mb.BaseMetricSet) (mb.MetricSet, error) {
+	cfgwarn.Experimental("The couchdb server metricset is experimental.")


I think we can go directly to beta for these metricsets, please update it in the fields files too.

metricbeat/module/couchdb/_meta/docs.asciidoc

jsoriano · 2019-01-09T13:29:08Z

metricbeat/module/couchdb/server/server_test.go

+	"github.com/stretchr/testify/assert"
+)
+
+const testFile = "../_meta/test/serverstats.json"


Not used variable.

jsoriano · 2019-01-09T13:29:55Z

metricbeat/module/couchdb/_meta/test/serverstats.json

@@ -0,0 +1,43 @@
+  "couchdb": {


Is this an incomplete JSON? It should start by {. The error in travis seems caused by this.

I may have overlooked this field. I made the correction. I think the error in travis isn't about this. CouchDB returns an error because database does not exist in Docker. This causes the TestFetch() methods in integration tests to fail. I'm trying to fix this error, but I haven't found a solution yet.

jsoriano · 2019-01-09T13:32:21Z

metricbeat/module/couchdb/server/data.go

+
+func eventMapping(content []byte) common.MapStr {
+	var data Server
+	json.Unmarshal(content, &data)


Check for errors here.

Thanks for your review. I believe I've addressed all the feedback now.

ruflin · 2019-01-10T08:12:49Z

metricbeat/module/couchdb/server/_meta/data.json

@@ -0,0 +1,58 @@
+{
+    "@timestamp":"2016-05-23T08:05:34.853Z",


As the indentation is off in this file and contains some old fields, I wonder if this was generated or manually created?

This was generated. Should I update this file?

Yes, can you generated it again to see if the strange indentation disappears?

ruflin · 2019-01-10T08:13:41Z

metricbeat/module/couchdb/server/_meta/fields.yml

+            Number of HTTP COPY requests
+
+        - name: HEAD
+          type: scaled_float


Should these header counters(?) be long instead of float?

ruflin · 2019-01-10T08:13:55Z

metricbeat/module/couchdb/server/_meta/fields.yml

+        HTTP status codes statistics
+      fields:
+        - name: "200"
+          type: scaled_float


Same question as above: long instead of float?

ruflin · 2019-01-10T08:14:09Z

metricbeat/module/couchdb/server/_meta/fields.yml

+        couchdb statistics
+      fields:
+        - name: database_writes
+          type: scaled_float


long? Same for most fields below.

I changed all fields with float to long.

ruflin · 2019-01-14T11:56:27Z

metricbeat/module/couchdb/server/server_integration_test.go

+}
+
+/*
+// TODO: Enable


This should be enabled again I assume?

…into couchdb-module

jsoriano · 2019-01-27T18:14:01Z

jenkins, test this

jsoriano · 2019-01-27T18:20:44Z

Thanks @berfinsari! I think we can merge this and continue with follow ups, I have created a new issue to keep track of development of this module (#10354).

jsoriano · 2019-01-27T18:42:05Z

metricbeat/tests/system/test_couchdb.py

+
+    def get_hosts(self):
+        return [os.getenv('COUCHDB_HOST', 'localhost') + ':' +
+                os.getenv('COUCHDB_PORT', '5894')]


Suggested change

os.getenv('COUCHDB_PORT', '5894')]

os.getenv('COUCHDB_PORT', '5984')]

To be fixed in follow up #10358

jsoriano · 2019-01-28T08:33:37Z

Merged, @berfinsari do you happen to have some dashboards you'd like to share too? 🙂

berfinsari added 2 commits December 5, 2018 22:54

Add couchdb module with server metricset

1fe3989

Add couchdb module

2a27d07

houndci-bot reviewed Dec 5, 2018

View reviewed changes

Adding entry to CHANGELOG

5f03dd8

ruflin added module review Metricbeat Metricbeat Team:Integrations Label for the Integrations team labels Dec 6, 2018

sayden self-assigned this Dec 6, 2018

Add comment to exported fields

fb6e802

alvarolobato unassigned sayden Dec 10, 2018

Run make update fmt command in x-pack/metricbeat directory

a091a56

berfinsari requested a review from a team as a code owner December 14, 2018 15:16

jsoriano requested changes Dec 14, 2018

View reviewed changes

ruflin reviewed Dec 17, 2018

View reviewed changes

berfinsari added 2 commits December 20, 2018 02:02

Add description of fields

1ed6361

Change couchdb version in Dockerfile

0da9615

Update generated docs

a8852e5

jsoriano reviewed Dec 20, 2018

View reviewed changes

sayden self-requested a review December 20, 2018 10:42

berfinsari requested review from a team as code owners January 8, 2019 18:33

Run make update fmt command in x-pack/metricbeat directory

26227f7

urso removed the request for review from a team January 8, 2019 22:20

jsoriano reviewed Jan 9, 2019

View reviewed changes

berfinsari added 4 commits January 10, 2019 02:39

Add module and metricset releases

cac74a7

Adding check for errors

362140d

Fix incomplete json in test files

ea89d1f

Update generated docs

c2ec79f

ruflin reviewed Jan 10, 2019

View reviewed changes

berfinsari added 2 commits January 12, 2019 01:18

Update server integration test

068136a

Change type float to long in fields.yml

ab38d19

ruflin reviewed Jan 14, 2019

View reviewed changes

berfinsari added 7 commits January 26, 2019 04:56

Update data.json

88563e0

Merge branch 'master' into couchdb-module

a744eff

Change Couchdb version and Update integration test

38d54e1

Merge branch 'couchdb-module' of https://github.com/berfinsari/beats …

af9bfd0

…into couchdb-module

Update fields.go

78baf23

Fix fields.go issue

8a30cb8

Fix not documented event in system test

d8eaf90

jsoriano mentioned this pull request Jan 27, 2019

CouchDB metribeat module #10354

Closed

14 tasks

jsoriano approved these changes Jan 27, 2019

View reviewed changes

jsoriano reviewed Jan 27, 2019

View reviewed changes

jsoriano merged commit 6293568 into elastic:master Jan 28, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add couchdb module #9406

Add couchdb module #9406

berfinsari commented Dec 5, 2018

elasticmachine commented Dec 5, 2018

elasticmachine commented Dec 6, 2018

berfinsari commented Dec 11, 2018

ruflin commented Dec 12, 2018

berfinsari commented Dec 13, 2018

jsoriano commented Dec 13, 2018

jsoriano left a comment

jsoriano commented Dec 14, 2018

ruflin Dec 17, 2018

berfinsari Dec 19, 2018

berfinsari commented Dec 19, 2018

jsoriano commented Dec 20, 2018

sayden commented Dec 20, 2018

jsoriano Dec 20, 2018

jsoriano Jan 9, 2019

jsoriano Jan 9, 2019

jsoriano Jan 9, 2019

berfinsari Jan 10, 2019

jsoriano Jan 9, 2019

berfinsari Jan 10, 2019

ruflin Jan 10, 2019

berfinsari Jan 11, 2019

ruflin Jan 14, 2019

ruflin Jan 10, 2019

ruflin Jan 10, 2019

ruflin Jan 10, 2019

berfinsari Jan 11, 2019

ruflin Jan 14, 2019

jsoriano commented Jan 27, 2019

jsoriano commented Jan 27, 2019

jsoriano Jan 27, 2019

jsoriano Jan 28, 2019

jsoriano commented Jan 28, 2019

	os.getenv('COUCHDB_PORT', '5894')]
	os.getenv('COUCHDB_PORT', '5984')]

Add couchdb module #9406

Add couchdb module #9406

Conversation

berfinsari commented Dec 5, 2018

elasticmachine commented Dec 5, 2018

elasticmachine commented Dec 6, 2018

berfinsari commented Dec 11, 2018

ruflin commented Dec 12, 2018

berfinsari commented Dec 13, 2018

jsoriano commented Dec 13, 2018

jsoriano left a comment

Choose a reason for hiding this comment

jsoriano commented Dec 14, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

berfinsari commented Dec 19, 2018

jsoriano commented Dec 20, 2018

sayden commented Dec 20, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsoriano commented Jan 27, 2019

jsoriano commented Jan 27, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsoriano commented Jan 28, 2019