Simple conversion of the format returned by the API

[ph]

DO NOT MERGE REQUIRE CODE CHANGE IN KIBANA

---

The Kibana API for CM is undergoing internal changes to better allow
future scalability of the configurations, one of the requirements for
this move was to flatten as much as possible the configuration file so
the format would be the same independant of the key that need to
received ot the configuration. It will be up to the manager to take the
JSON format and convert it into something that our Configuration can
understand.

ref: elastic/kibana#27717

---

Details about the implementation:

Sorry, It took much longer than I thought it would, and I initially wanted to do it more cleanly and allow the reloadable registry to tells us what it actually needed. I was actually pretty far into it but after taking a step back I was anxious about my strategy and the number of required changes and the implication in review that I've decided to go for a simple take on it instead.

Right now the registry as two concepts a Single and List, I think we should uniform them into a single Reloable and let the implementation details manage how a reload is happening and we also need better feedback from the reload part.

[ph]

@mattapperson Concerning the id that will be returned for each config_block will Where it will be located at the same level of the type of in the config?

{
    "configuration_blocks": [

        {
            "config": {
              "sub_type": "elasticsearch"
              "hosts": [
                  "localhost"
              ],
              "password": "foobar"
              "username": "elastic"
            },
            "type": "output"
            "id": "12312341231231"
        }
    ]
}

{
    "configuration_blocks": [

        {
            "config": {
              "sub_type": "elasticsearch"
              "id": "12312341231231"
              "hosts": [
                  "localhost"
              ],
              "password": "foobar"
              "username": "elastic"
            },
            "type": "output"

        }
    ]
}

[ph]

Winlogbeat failure on beats-ci is due to a network issue fetching the golang artifact.

[mattapperson]

sub_type should be _sub_type where the prefixed _ ensures its known to be "special"

[ph]

@mattapperson Concerning the ID, its currently possible in hearthbeat to define an ID for the current module, So I think our config block id that you will generate should probably be:

1. Either defined outside of config.
2. Or prefixed with an underscore to make sure we do not have any conflict with module or config human defined fields.

[urso]

Just from reading the code it is not really clear why the converters are needed. Maybe we should also include a doc.go file explaining the API/protocol?

[ph]

I will add a doc illustrating the before and after.

[ph]

I've enabled the tests in this PR again and also fixed the things from the review.

@mattapperson what is the value of _sub_type when we define a normal filebeat input?

[mattapperson]

@ph in such a case it is not defined. _sub_type is optional, and used when the value needs to be both nested and the top level of the nest needs to be dynamic.

[ph]

I tested this PR with elastic/kibana#27717, I haven't found any issue with the conversion, so once the other PR is merged we can merge this one also. However, I have added a few comment on the kibana side that need to be fixed before the refactor is merged.

[ph]

This will need to go in 6.x

[codingogre]

Has the upgrade paths been tested with this change? Upgrading from past versions to 6.7 includes this code, but looking at the .management-beats index after registering a beat looks like this:

{
  "_index": ".management-beats",
  "_type": "_doc",
  "_id": "tag:Filebeat-System",
  "_version": 3,
  "_score": 0,
  "_source": {
    "tag": {
      "color": "#00b3a4",
      "configuration_blocks": [
        {
          "type": "filebeat.modules",
          "configs": [
            {
              "module": "system"
            }
          ]
        }
      ],
      "id": "Filebeat-System",
      "last_updated": "2019-04-03T23:30:42.137Z"
    },
    "type": "tag"
  },
  "fields": {
    "tag.last_updated": [
      "2019-04-03T23:30:42.137Z"
    ]
  }
}

This generates errors in the filebeat (v6.7) log and results in filebeat not being able to pick up new configuration changes which totally breaks Centralized Beats Management.

2019-04-04T02:56:44.820-0400 ERROR [centralmgmt] management/manager.go:224 error retrieving new configurations, will use cached ones: error unmarshaling Kibana response: '_sub_type' key not found

Looking through the code it seems there are tests where a new config type is used:

From beats/x-pack/libbeat/management/api/configuration_test.go

{
	"type": "filebeat.modules",
	"config": {
		"_sub_type": "system"
	}
}

[ph]

@codingogre did you run the migration tool ? https://github.com/elastic/migrate-management-beats/