Skip to content

Commit

Permalink
rename eventsLogger to sensitiveLogger
Browse files Browse the repository at this point in the history
  • Loading branch information
belimawr committed Jan 22, 2024
1 parent 3d389d9 commit f8253aa
Show file tree
Hide file tree
Showing 51 changed files with 166 additions and 165 deletions.
4 changes: 2 additions & 2 deletions auditbeat/auditbeat.reference.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1553,14 +1553,14 @@ logging.files:

# Having a different log file for raw events also prevents event data
# from drowning out the regular log files.
#logging.events:
#logging.sensitive:
#files:
# Configure the path where the logs are written. The default is the logs directory
# under the home path (the binary location).
#path: /var/log/auditbeat

# The name of the files where the logs are written to.
#name: auditbeat-events-data
#name: auditbeat-sensitive

# Configure log file size limit. If the limit is reached, log file will be
# automatically rotated.
Expand Down
4 changes: 2 additions & 2 deletions auditbeat/auditbeat.yml
Original file line number Diff line number Diff line change
Expand Up @@ -174,14 +174,14 @@ processors:
# log messages, a different log file, only for log entries containing raw events,
# is used. It will use the same level, selectors and all other configurations
# from the default logger, but it will have it's own file configuration.
#logging.events:
#logging.sensitive:
#files:
# Configure the path where the logs are written. The default is the logs directory
# under the home path (the binary location).
#path: /var/log/auditbeat

# The name of the files where the logs are written to.
#name: auditbeat-events-data
#name: auditbeat-sensitive

# ============================= X-Pack Monitoring ==============================
# Auditbeat can export internal metrics to a central Elasticsearch monitoring
Expand Down
4 changes: 2 additions & 2 deletions filebeat/filebeat.reference.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2649,14 +2649,14 @@ logging.files:

# Having a different log file for raw events also prevents event data
# from drowning out the regular log files.
#logging.events:
#logging.sensitive:
#files:
# Configure the path where the logs are written. The default is the logs directory
# under the home path (the binary location).
#path: /var/log/filebeat

# The name of the files where the logs are written to.
#name: filebeat-events-data
#name: filebeat-sensitive

# Configure log file size limit. If the limit is reached, log file will be
# automatically rotated.
Expand Down
4 changes: 2 additions & 2 deletions filebeat/filebeat.yml
Original file line number Diff line number Diff line change
Expand Up @@ -191,14 +191,14 @@ processors:
# log messages, a different log file, only for log entries containing raw events,
# is used. It will use the same level, selectors and all other configurations
# from the default logger, but it will have it's own file configuration.
#logging.events:
#logging.sensitive:
#files:
# Configure the path where the logs are written. The default is the logs directory
# under the home path (the binary location).
#path: /var/log/filebeat

# The name of the files where the logs are written to.
#name: filebeat-events-data
#name: filebeat-sensitive

# ============================= X-Pack Monitoring ==============================
# Filebeat can export internal metrics to a central Elasticsearch monitoring
Expand Down
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -420,4 +420,4 @@ replace (
// Exclude this version because the version has an invalid checksum.
exclude github.com/docker/distribution v2.8.0+incompatible

replace github.com/elastic/elastic-agent-libs => github.com/belimawr/elastic-agent-libs v0.2.9-0.20240116105334-25f61a14ad41
replace github.com/elastic/elastic-agent-libs => github.com/belimawr/elastic-agent-libs v0.2.9-0.20240122163001-efb117578ab2
4 changes: 2 additions & 2 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -373,8 +373,8 @@ github.com/awslabs/goformation/v4 v4.1.0 h1:JRxIW0IjhYpYDrIZOTJGMu2azXKI+OK5dP56
github.com/awslabs/goformation/v4 v4.1.0/go.mod h1:MBDN7u1lMNDoehbFuO4uPvgwPeolTMA2TzX1yO6KlxI=
github.com/awslabs/kinesis-aggregation/go/v2 v2.0.0-20220623125934-28468a6701b5 h1:lxW5Q6K2IisyF5tlr6Ts0W4POGWQZco05MJjFmoeIHs=
github.com/awslabs/kinesis-aggregation/go/v2 v2.0.0-20220623125934-28468a6701b5/go.mod h1:0Qr1uMHFmHsIYMcG4T7BJ9yrJtWadhOmpABCX69dwuc=
github.com/belimawr/elastic-agent-libs v0.2.9-0.20240116105334-25f61a14ad41 h1:4kwfzIBmNATT0es3HsgZP7W4p6OUo1TCOk5qchsUzTs=
github.com/belimawr/elastic-agent-libs v0.2.9-0.20240116105334-25f61a14ad41/go.mod h1:pGMj5myawdqu+xE+WKvM5FQzKQ/MonikkWOzoFTJxaU=
github.com/belimawr/elastic-agent-libs v0.2.9-0.20240122163001-efb117578ab2 h1:QOTo5kTJ8oqdrSOH8/OhSkEMA3mnRltGg52M9YyH7Zo=
github.com/belimawr/elastic-agent-libs v0.2.9-0.20240122163001-efb117578ab2/go.mod h1:pGMj5myawdqu+xE+WKvM5FQzKQ/MonikkWOzoFTJxaU=
github.com/benbjohnson/clock v1.1.0/go.mod h1:J11/hYXuz8f4ySSvYwY0FKfm+ezbsZBKZxNJlLklBHA=
github.com/benbjohnson/immutable v0.2.1/go.mod h1:uc6OHo6PN2++n98KHLxW8ef4W42ylHiQSENghE1ezxI=
github.com/benbjohnson/tmpl v1.0.0/go.mod h1:igT620JFIi44B6awvU9IsDhR77IXWtFigTLil/RPdps=
Expand Down
4 changes: 2 additions & 2 deletions heartbeat/heartbeat.reference.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1645,14 +1645,14 @@ logging.files:

# Having a different log file for raw events also prevents event data
# from drowning out the regular log files.
#logging.events:
#logging.sensitive:
#files:
# Configure the path where the logs are written. The default is the logs directory
# under the home path (the binary location).
#path: /var/log/heartbeat

# The name of the files where the logs are written to.
#name: heartbeat-events-data
#name: heartbeat-sensitive

# Configure log file size limit. If the limit is reached, log file will be
# automatically rotated.
Expand Down
4 changes: 2 additions & 2 deletions heartbeat/heartbeat.yml
Original file line number Diff line number Diff line change
Expand Up @@ -157,14 +157,14 @@ processors:
# log messages, a different log file, only for log entries containing raw events,
# is used. It will use the same level, selectors and all other configurations
# from the default logger, but it will have it's own file configuration.
#logging.events:
#logging.sensitive:
#files:
# Configure the path where the logs are written. The default is the logs directory
# under the home path (the binary location).
#path: /var/log/heartbeat

# The name of the files where the logs are written to.
#name: heartbeat-events-data
#name: heartbeat-sensitive

# ============================= X-Pack Monitoring ==============================
# Heartbeat can export internal metrics to a central Elasticsearch monitoring
Expand Down
4 changes: 2 additions & 2 deletions libbeat/_meta/config/logging.reference.yml.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -77,14 +77,14 @@ logging.files:

# Having a different log file for raw events also prevents event data
# from drowning out the regular log files.
#logging.events:
#logging.sensitive:
#files:
# Configure the path where the logs are written. The default is the logs directory
# under the home path (the binary location).
#path: /var/log/{{.BeatName}}

# The name of the files where the logs are written to.
#name: {{.BeatName}}-events-data
#name: {{.BeatName}}-sensitive

# Configure log file size limit. If the limit is reached, log file will be
# automatically rotated.
Expand Down
4 changes: 2 additions & 2 deletions libbeat/_meta/config/logging.yml.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -14,11 +14,11 @@
# log messages, a different log file, only for log entries containing raw events,
# is used. It will use the same level, selectors and all other configurations
# from the default logger, but it will have it's own file configuration.
#logging.events:
#logging.sensitive:
#files:
# Configure the path where the logs are written. The default is the logs directory
# under the home path (the binary location).
#path: /var/log/{{.BeatName}}

# The name of the files where the logs are written to.
#name: {{.BeatName}}-events-data
#name: {{.BeatName}}-sensitive
61 changes: 31 additions & 30 deletions libbeat/cmd/instance/beat.go
Original file line number Diff line number Diff line change
Expand Up @@ -122,15 +122,15 @@ type beatConfig struct {
Features *config.C `config:"features"`

// beat internal components configurations
HTTP *config.C `config:"http"`
HTTPPprof *pprof.Config `config:"http.pprof"`
BufferConfig *config.C `config:"http.buffer"`
Path paths.Path `config:"path"`
Logging *config.C `config:"logging"`
EventLogging *config.C `config:"logging.events"`
MetricLogging *config.C `config:"logging.metrics"`
Keystore *config.C `config:"keystore"`
Instrumentation instrumentation.Config `config:"instrumentation"`
HTTP *config.C `config:"http"`
HTTPPprof *pprof.Config `config:"http.pprof"`
BufferConfig *config.C `config:"http.buffer"`
Path paths.Path `config:"path"`
Logging *config.C `config:"logging"`
SensitiveLogging *config.C `config:"logging.sensitive"`
MetricLogging *config.C `config:"logging.metrics"`
Keystore *config.C `config:"keystore"`
Instrumentation instrumentation.Config `config:"instrumentation"`

// output/publishing related configurations
Pipeline pipeline.Config `config:",inline"`
Expand Down Expand Up @@ -386,29 +386,30 @@ func (b *Beat) createBeater(bt beat.Creator) (beat.Beater, error) {
// we need some defaults to be populates otherwise Unpack will
// fail. We also overwrite some defaults that are specific to the
// events logger.
eventsLoggerCfg := logp.DefaultConfig(configure.GetEnvironment())
eventsLoggerCfg.Files.MaxSize = 5242880 // 5MB
eventsLoggerCfg.Files.MaxBackups = 5
sensitiveLoggerCfg := logp.DefaultConfig(configure.GetEnvironment())
sensitiveLoggerCfg.ToFiles = true // make the default explicit
sensitiveLoggerCfg.Files.MaxSize = 5242880 // 5MB
sensitiveLoggerCfg.Files.MaxBackups = 5

// merge eventsLoggerCfg with b.Config.Logging, so logging.events.* only
// merge sensitiveLoggerCfg with b.Config.Logging, so logging.sensitive.* only
// overwrites the files block.
if err := b.Config.EventLogging.Unpack(&eventsLoggerCfg); err != nil {
if err := b.Config.SensitiveLogging.Unpack(&sensitiveLoggerCfg); err != nil {
return nil, fmt.Errorf("error initialising events logger: %w", err)
}

// Ensure the default filename is set
if eventsLoggerCfg.Files.Name == "" {
eventsLoggerCfg.Files.Name = b.Info.Beat
if sensitiveLoggerCfg.Files.Name == "" {
sensitiveLoggerCfg.Files.Name = b.Info.Beat
// Append the name so the files do not overwrite themselves.
eventsLoggerCfg.Files.Name = eventsLoggerCfg.Files.Name + "-events-data"
sensitiveLoggerCfg.Files.Name = sensitiveLoggerCfg.Files.Name + "-events-data"
}

// Now that the events logger is configured, we can register it's diagnostic
// hook
b.Manager.RegisterDiagnosticHook("events log",
"log files containing raw events", "events_log.ndjson",
"application/x-ndjson", b.eventsLogDiagnosticsHook(eventsLoggerCfg))
outputFactory := b.makeOutputFactory(b.Config.Output, eventsLoggerCfg)
"application/x-ndjson", b.eventsLogDiagnosticsHook(sensitiveLoggerCfg))
outputFactory := b.makeOutputFactory(b.Config.Output, sensitiveLoggerCfg)
settings := pipeline.Settings{
Processors: b.processors,
InputQueueSize: b.InputQueueSize,
Expand All @@ -418,7 +419,7 @@ func (b *Beat) createBeater(bt beat.Creator) (beat.Beater, error) {
return nil, fmt.Errorf("error initializing publisher: %w", err)
}

reload.RegisterV2.MustRegisterOutput(b.makeOutputReloader(publisher.OutputReloader(), eventsLoggerCfg))
reload.RegisterV2.MustRegisterOutput(b.makeOutputReloader(publisher.OutputReloader(), sensitiveLoggerCfg))

// TODO: some beats race on shutdown with publisher.Stop -> do not call Stop yet,
// but refine publisher to disconnect clients on stop automatically
Expand Down Expand Up @@ -855,18 +856,18 @@ func (b *Beat) configure(settings Settings) error {
// If either b.Config.EventLoggingor b.Config.Logging are nil
// merging them will fail, so in case any of them is nil,
// we set them to an empty config.C
if b.Config.EventLogging == nil {
b.Config.EventLogging = config.NewConfig()
if b.Config.SensitiveLogging == nil {
b.Config.SensitiveLogging = config.NewConfig()
}
if b.Config.Logging == nil {
b.Config.Logging = config.NewConfig()
}
if err := b.Config.EventLogging.Merge(b.Config.Logging); err != nil {
return fmt.Errorf("cannot merge logging and logging.events configuration: %w", err)
if err := b.Config.SensitiveLogging.Merge(b.Config.Logging); err != nil {
return fmt.Errorf("cannot merge logging and logging.sensitive configuration: %w", err)
}

if _, err := b.Config.EventLogging.Remove("events", -1); err != nil {
return fmt.Errorf("cannot update logging.events configuration: %w", err)
if _, err := b.Config.SensitiveLogging.Remove("events", -1); err != nil {
return fmt.Errorf("cannot update logging.sensitive configuration: %w", err)
}

if err := promoteOutputQueueSettings(&b.Config); err != nil {
Expand Down Expand Up @@ -1176,7 +1177,7 @@ func (b *Beat) indexSetupCallback() elasticsearch.ConnectCallback {
}
}

func (b *Beat) makeOutputReloader(outReloader pipeline.OutputReloader, eventsLoggerCfg logp.Config) reload.Reloadable {
func (b *Beat) makeOutputReloader(outReloader pipeline.OutputReloader, sensitiveLoggerCfg logp.Config) reload.Reloadable {
return reload.ReloadableFunc(func(update *reload.ConfigWithMeta) error {
if update == nil {
return nil
Expand All @@ -1198,7 +1199,7 @@ func (b *Beat) makeOutputReloader(outReloader pipeline.OutputReloader, eventsLog
}
}

return outReloader.Reload(update, eventsLoggerCfg, b.createOutput)
return outReloader.Reload(update, sensitiveLoggerCfg, b.createOutput)
})
}

Expand Down Expand Up @@ -1303,7 +1304,7 @@ func (b *Beat) reloadOutputOnCertChange(cfg config.Namespace) error {
return nil
}

func (b *Beat) createOutput(stats outputs.Observer, cfg config.Namespace, eventsLoggerCfg logp.Config) (outputs.Group, error) {
func (b *Beat) createOutput(stats outputs.Observer, cfg config.Namespace, sensitiveLoggerCfg logp.Config) (outputs.Group, error) {
if !cfg.IsSet() {
return outputs.Group{}, nil
}
Expand All @@ -1312,7 +1313,7 @@ func (b *Beat) createOutput(stats outputs.Observer, cfg config.Namespace, events
return outputs.Group{}, fmt.Errorf("could not setup output certificates reloader: %w", err)
}

return outputs.Load(b.IdxSupporter, b.Info, stats, cfg.Name(), cfg.Config(), eventsLoggerCfg)
return outputs.Load(b.IdxSupporter, b.Info, stats, cfg.Name(), cfg.Config(), sensitiveLoggerCfg)
}

func (b *Beat) registerClusterUUIDFetching() {
Expand Down
4 changes: 2 additions & 2 deletions libbeat/cmd/instance/beat_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -267,8 +267,8 @@ type outputReloaderMock struct {

func (r *outputReloaderMock) Reload(
cfg *reload.ConfigWithMeta,
eventsLoggerCfg logp.Config,
factory func(o outputs.Observer, cfg config.Namespace, eventsLoggerCfg logp.Config) (outputs.Group, error),
sensitiveLoggerCfg logp.Config,
factory func(o outputs.Observer, cfg config.Namespace, sensitiveLoggerCfg logp.Config) (outputs.Group, error),
) error {
r.cfg = cfg
return nil
Expand Down
20 changes: 10 additions & 10 deletions libbeat/docs/loggingconfig.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -296,7 +296,7 @@ Below are some samples:

ifndef::serverless[]
[float]
=== Configuration options for events logger
=== Configuration options for sensitive logger

Some outputs will log raw events on errors like indexing errors in the
Elasticsearch output, to prevent logging raw events (that may contain
Expand All @@ -309,34 +309,34 @@ Having a different log file for raw events also prevents event data
from drowning out the regular log files.

IMPORTANT: No matter the default logger output configuration, raw events
will **always** be logged to a file configured by `logging.events.files`.
will **always** be logged to a file configured by `logging.sensitive.files`.

[float]
==== `logging.events.files.path`
==== `logging.sensitive.files.path`

The directory that log files are written to. The default is the logs path. See
the <<directory-layout>> section for details.

[float]
==== `logging.events.files.name`
==== `logging.sensitive.files.name`

The name of the file that logs are written to. The default is '{beatname_lc}'.
The name of the file that logs are written to. The default is '{beatname_lc}'-sensitive.

[float]
==== `logging.events.files.rotateeverybytes`
==== `logging.sensitive.files.rotateeverybytes`

The maximum size of a log file. If the limit is reached, a new log file is
generated. The default size limit is 5242880 (5 MB).

[float]
==== `logging.events.files.keepfiles`
==== `logging.sensitive.files.keepfiles`

The number of most recent rotated log files to keep on disk. Older files are
deleted during log rotation. The default value is 5. The `keepfiles` options has
to be in the range of 2 to 1024 files.

[float]
==== `logging.events.files.permissions`
==== `logging.sensitive.files.permissions`

The permissions mask to apply when rotating log files. The default value is
0600. The `permissions` option must be a valid Unix-style file permissions mask
Expand All @@ -354,7 +354,7 @@ Examples:
* 0600: give read and write access to the file owner, and no access to all others.

[float]
==== `logging.events.files.interval`
==== `logging.sensitive.files.interval`

Enable log file rotation on time intervals in addition to size-based rotation.
Intervals must be at least 1s. Values of 1m, 1h, 24h, 7*24h, 30*24h, and 365*24h
Expand All @@ -363,7 +363,7 @@ reported by the local system clock. All other intervals are calculated from the
unix epoch. Defaults to disabled.

[float]
==== `logging.events.files.rotateonstartup`
==== `logging.sensitive.files.rotateonstartup`

If the log file already exists on startup, immediately rotate it and start
writing to a new file instead of appending to the existing one. Defaults to
Expand Down
2 changes: 1 addition & 1 deletion libbeat/outputs/console/console.go
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ func makeConsole(
beat beat.Info,
observer outputs.Observer,
cfg *config.C,
eventsLoggerCfg logp.Config,
sensitiveLoggerCfg logp.Config,
) (outputs.Group, error) {
config := defaultConfig
err := cfg.Unpack(&config)
Expand Down
Loading

0 comments on commit f8253aa

Please sign in to comment.