Beatless has now his default list of settings #8452
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Beatless initial PR Add a new beat inside the x-pack folder under the Elastic License, minimal requirement changes to have a build and a test running. Main makefile exclude ASL2 for x-pack but check for Elastic. Beats can override the license in their Makefile.
In beat each data collector need to initialize his own beat.Client to
have access to the pipeline. The current pipeline implementation is
completely asynchronous, meaning when you publish something to the
queue, you don't know if it will be send or when it will be send.
Some system like aws lambda requires to be in sync, when the method
return we expect the events to be send. This PR allow to change the
behavior to have a sync publish that leverage the pipeline callbacks.
Notes: it also changes the client interface, since publish and publishAll can
return an error.
Usage:
```
sc, err := NewSyncClient(pipeline, beat.ClientConfig{})
if !assert.NoError(t, err) {
return
}
err := sc.PublishAll()
if err != nil {
...
}
sc.Wait() // block until the publish is done.
defer sc.Close() // this call will also block
```
* License manager Implements a License manager inside beats, as we development more features that depends on the licensing and the capabilities of a remote cluster we need a unique way to access that information. This commit implements the following: Add a License manager that can be started at the beginning of the beats instance initialization. The manager takes a fetcher, currently we only support Elasticsearch as the license backend but we could add support for an Logstash endpoint that could proxy the license. Notes: - By default when the manager is started, no license is available, calling `Get()` on the manager will return a license not found. - The manager will periodically retrieve the license from the fetcher. - When an error occurs on the periodic check, the license wont be invalidated right away but will enter a grace period, after this period the license will be invalidated and will replaced by the OSS license. - License and capabilities and be retrieved by calling `Get()` or registering a type implementing the `Watcher` interface.
This provide the following: 1. Plugin infra for developing providers and functions 2. A local stdin provider only used for testing, I will remove it in the final version. 3. AWS provider and function types for: - Cloudwatch logs - SQS - Kinesis - Api web gateway proxy 4. License checker 5. Packaging of artifact 6. Runners 7. CLI infrastructure 8. CLI to push a cloudwatch logs function. 9. CLI to delete any function 10. Processors support. What it doesn't provides: - ECS and full event extraction. - Specifying the AWS credentials in the configuration - CLI for SQS, Kinesis, API - Robust CLI interaction with the API, rollback on failure / versioning. - Removal of not supported outputs - Removal of seccomp check - Integration tests - Updated build task to produce containing the user executable beat and the linux beats. Vendored: - https://github.com/aws/aws-lambda-go - https://github.com/aws/aws-sdk-go-v2
When a project had two licenses, the script was creating two entries in the NOTICE.txt but both entries would share the same license. This commit also add the possibility to skip file that start with LICENSE but doesn't contains any license information. It also add MIT-0 license wording see https://github.com/aws/mit-0
Sometime a custom beat that get executed want to override system defaults instead of relying on code defined by libbeat. This is the case with beatless, the queue has limits and flush values that make sense when the beat is run on the edge but not on on AWS lambda. Note that settings is passed via liberal common.Config no type checking is done. I went that route because many parts of beats doesn't expose the config struct outside of the package. This is similar to explicitely defining them in the yaml.
instead of relying on libbeat default beatless can now overrides default settings for outputs or queues.
houndci-bot
reviewed
Sep 26, 2018
| // feature are grouped by namespace, a namespace is a kind of plugin like outputs, inputs, or queue. | ||
| // The feature name must be unique. | ||
| type registry struct { | ||
| type FeatureRegistry struct { |
There was a problem hiding this comment.
type name will be used as feature.FeatureRegistry by other packages, and that stutters; consider calling this Registry
| @@ -28,26 +28,26 @@ import ( | |||
|
|
|||
| type mapper map[string]map[string]Featurable | |||
|
|
|||
| // Registry implements a global registry for any kind of feature in beats. | |||
| // Registry implements a global FeatureRegistry for any kind of feature in beats. | |||
There was a problem hiding this comment.
comment on exported type FeatureRegistry should be of the form "FeatureRegistry ..." (with optional leading article)
|
changed from review to in progress because we have to get the other big one merged. |
ph
force-pushed
the
feature-beatless
branch
from
fd0a33e to
533ba8c
Compare
|
moved to #8437 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
instead of relying on libbeat default beatless can now overrides default settings
for outputs or queues.
depends on #8449