10 Mar 14:43

40e633f

syslog-ng-4.1.1

4.1.1

This is the combination of the news entries of 4.1.0 and 4.1.1.
4.1.1 hotfixed a grouping-by() and db-parser() related crash.

Highlights

PROXY protocol v2 support (#4211)

We've added support for PROXY protocol v2 (transport(proxied-tcp)), a protocol
used by network load balancers, such as Amazon Elastic Load Balancer and
HAProxy, to carry original source/destination address information, as described
in https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt

Metrics revised

Prometheus metric format (#4325)

A new metric system has been introduced to syslog-ng, where metrics are
identified by names and partitioned by labels, which is similar to the
Prometheus data model.

The syslog-ng-ctl stats prometheus command can be used to query syslog-ng
metrics in a format that conforms to the Prometheus text-based exposition
format.

syslog-ng-ctl stats prometheus --with-legacy-metrics displays legacy metrics
as well. Legacy metrics do not follow Prometheus' metric and label conventions.

Classification (metadata-based metrics) (#4318)

metrics-probe(), a new parser has also been added, which counts messages
passing through based on the metadata of each message. The parser creates
labeled metrics based on the fields of the message.

Both the key and labels can be set in the config, the values of the labels can
be templated. E.g.:

parser p_metrics_probe {
  metrics-probe(
    key("custom_key")  # adds "syslogng_" prefix => "syslogng_custom_key"
    labels(
      "custom_label_name_1" => "foobar"
      "custom_label_name_2" => "${.custom.field}"
    )
  );
};

With this config, it creates counters like these:

syslogng_custom_key{custom_label_name_1="foobar", custom_label_name_2="bar"} 1
syslogng_custom_key{custom_label_name_1="foobar", custom_label_name_2="foo"} 1
syslogng_custom_key{custom_label_name_1="foobar", custom_label_name_2="baz"} 3

The minimal config creates counters with the key
syslogng_classified_events_total and labels app, host, program and
source. E.g.:

parser p_metrics_probe {
  metrics-probe();
};

With this config, it creates counters like these:

syslogng_classified_events_total{app="example-app", host="localhost", program="baz", source="s_local_1"} 3
syslogng_classified_events_total{app="example-app", host="localhost", program="bar", source="s_local_1"} 1
syslogng_classified_events_total{app="example-app", host="localhost", program="foo", source="s_local_1"} 1

Named log paths (path ingress/egress metrics) (#4344)

It is also possible to create named log paths, for example:

log top-level {
    source(s_local);

    log inner-1 {
        filter(f_inner_1);
        destination(d_local_1);
    };

    log inner-2 {
        filter(f_inner_2);
        destination(d_local_2);
    };
};

Each named log path counts its ingress and egress messages:

syslogng_log_path_ingress{id="top-level"} 114
syslogng_log_path_ingress{id="inner-1"} 114
syslogng_log_path_ingress{id="inner-2"} 114
syslogng_log_path_egress{id="top-level"} 103
syslogng_log_path_egress{id="inner-1"} 62
syslogng_log_path_egress{id="inner-2"} 41

Note that the egress statistics only count the messages which have been have not
been filtered out from the related log path, it does care about whether there
are any destinations in it or that any destination delivers or drops the
message.

The above three features are experimental; the output of stats prometheus
(names, labels, etc.) and the metrics created by metrics-probe() and named log
paths may change in the next 2-3 releases.

Features

$(format-date): add a new template function to format time and date values

$(format-date [options] format-string [timestamp])

$(format-date) takes a timestamp in the DATETIME representation and
formats it according to an strftime() format string. The DATETIME
representation in syslog-ng is a UNIX timestamp formatted as a decimal
number, with an optional fractional part, where the seconds and the
fraction of seconds are separated by a dot.

If the timestamp argument is missing, the timestamp of the message is
used.

Options:
--time-zone <TZstring> -- override timezone of the original timestamp
(#4202)
syslog-parser() and all syslog related sources: accept unquoted RFC5424
SD-PARAM-VALUEs instead of rejecting them with a parse error.

sdata-parser(): this new parser allows you to parse an RFC5424 style
structured data string. It can be used to parse this relatively complex
format separately.
(#4281)
system() source: the system() source was changed on systemd platforms to
fetch journal messages that relate to the current boot only (e.g. similar
to journalctl -fb) and to ignore messages generated in previous boots,
even if those messages were succesfully stored in the journal and were not
picked up by syslog-ng. This change was implemented as the journald access
APIs work incorrectly if time goes backwards across reboots, which is an
increasingly frequent event in virtualized environments and on systems that
lack an RTC. If you want to retain the old behaviour, please bypass the
system() source and use systemd-journal() directly, where this option
can be customized. The change is not tied to @version as we deemed the new
behaviour fixing an actual bug. For more information consult #2836.

systemd-journald() source: add match-boot() and matches() options to
allow you to constrain the collection of journal records to a subset of what
is in the journal. match-boot() is a yes/no value that allows you to fetch
messages that only relate to the current boot. matches() allows you to
specify one or more filters on journal fields.

Examples:
```
source s_journal_current_boot_only {
  systemd-source(match-boot(yes));
};

source s_journal_systemd_only {
  systemd-source(matches(
    "_COMM" => "systemd"
    )
  );
};
```
(#4245)
date-parser(): add value() parameter to instruct date-parser() to store
the resulting timestamp in a name-value pair, instead of changing the
timestamp value of the LogMessage.

datetime type representation: typed values in syslog-ng are represented as
strings when stored as a part of a log message. syslog-ng simply remembers
the type it was stored as. Whenever the value is used as a specific type in
a type-aware context where we need the value of the specific type, an
automatic string parsing takes place. This parsing happens for instance
whenever syslog-ng stores a datetime value in MongoDB or when
$(format-date) template function takes a name-value pair as parameter.
The datetime() type has stored its value as the number of milliseconds since
the epoch (1970-01-01 00:00:00 GMT). This has now been enhanced by making
it possible to store timestamps up to nanosecond resolutions along with an
optional timezone offset.

$(format-date): when applied to name-value pairs with the datetime type,
use the timezone offset if one is available.
(#4319)
stats: Added syslog-stats() global stats() group option.

E.g.:
```
options {
  stats(
    syslog-stats(no);
  );
};
```
It changes the behavior of counting messages based on different syslog-proto fields,
like SEVERITY, FACILITY, HOST, etc...

Possible values are:
- yes => force enable
- no => force disable
- auto => let stats(level()) decide (old behavior)
  (#4337)
kubernetes source: Added key-delimiter() option.

Some metadata fields can contain .-s in their name. This does not work
with syslog-ng-s macros, which by default use . as a delimiter. The added
key-delimiter() option changes this behavior by storing the parsed
metadata fields with a custom delimiter. In order to reach the fields, the
accessor side has to use the new delimiter format, e.g. --key-delimiter
option in $(format-json).
(#4213)

Bugfixes

Fix conditional evaluation with a dangling filter

We've fixed a bug that caused conditional evaluation (if/else/elif) and certain logpath flags (final, fallback)
to occasionally malfunction. The issue only happened in certain logpath constructs; examples can be found in the
PR description.
(#4058)
python: Fixed a bug, where PYTHONPATH was ignored with python3.11.
(#4298)
disk-buffer: Fixed disk-queue file becoming corrupt when changing disk-buf-size().

syslog-ng now continues with the originally set disk-buf-size().
Note that changing the disk-buf-size() of an existing disk-queue was never supported,
but could cause errors, which are fixed now.
(#4308)
dqtool: fix dqtool assign
([#4355](https://github.com/sys...

Assets 3

08 Mar 08:42

kira-syslogng

syslog-ng-4.1.0

1ca36dd

syslog-ng-4.1.0

4.1.0

Highlights

PROXY protocol v2 support (#4211)

Metrics revised

Prometheus metric format (#4325)

A new metric system has been introduced to syslog-ng, where metrics are
identified by names and partitioned by labels, which is similar to the
Prometheus data model.

The syslog-ng-ctl stats prometheus command can be used to query syslog-ng
metrics in a format that conforms to the Prometheus text-based exposition
format.

syslog-ng-ctl stats prometheus --with-legacy-metrics displays legacy metrics
as well. Legacy metrics do not follow Prometheus' metric and label conventions.

Classification (metadata-based metrics) (#4318)

Both the key and labels can be set in the config, the values of the labels can
be templated. E.g.:

parser p_metrics_probe {
  metrics-probe(
    key("custom_key")  # adds "syslogng_" prefix => "syslogng_custom_key"
    labels(
      "custom_label_name_1" => "foobar"
      "custom_label_name_2" => "${.custom.field}"
    )
  );
};

With this config, it creates counters like these:

syslogng_custom_key{custom_label_name_1="foobar", custom_label_name_2="bar"} 1
syslogng_custom_key{custom_label_name_1="foobar", custom_label_name_2="foo"} 1
syslogng_custom_key{custom_label_name_1="foobar", custom_label_name_2="baz"} 3

The minimal config creates counters with the key
syslogng_classified_events_total and labels app, host, program and
source. E.g.:

parser p_metrics_probe {
  metrics-probe();
};

With this config, it creates counters like these:

syslogng_classified_events_total{app="example-app", host="localhost", program="baz", source="s_local_1"} 3
syslogng_classified_events_total{app="example-app", host="localhost", program="bar", source="s_local_1"} 1
syslogng_classified_events_total{app="example-app", host="localhost", program="foo", source="s_local_1"} 1

Named log paths (path ingress/egress metrics) (#4344)

It is also possible to create named log paths, for example:

log top-level {
    source(s_local);

    log inner-1 {
        filter(f_inner_1);
        destination(d_local_1);
    };

    log inner-2 {
        filter(f_inner_2);
        destination(d_local_2);
    };
};

Each named log path counts its ingress and egress messages:

syslogng_log_path_ingress{id="top-level"} 114
syslogng_log_path_ingress{id="inner-1"} 114
syslogng_log_path_ingress{id="inner-2"} 114
syslogng_log_path_egress{id="top-level"} 103
syslogng_log_path_egress{id="inner-1"} 62
syslogng_log_path_egress{id="inner-2"} 41

The above three features are experimental; the output of stats prometheus
(names, labels, etc.) and the metrics created by metrics-probe() and named log
paths may change in the next 2-3 releases.

Features

$(format-date): add a new template function to format time and date values

$(format-date [options] format-string [timestamp])

$(format-date) takes a timestamp in the DATETIME representation and
formats it according to an strftime() format string. The DATETIME
representation in syslog-ng is a UNIX timestamp formatted as a decimal
number, with an optional fractional part, where the seconds and the
fraction of seconds are separated by a dot.

If the timestamp argument is missing, the timestamp of the message is
used.

Options:
--time-zone <TZstring> -- override timezone of the original timestamp
(#4202)
syslog-parser() and all syslog related sources: accept unquoted RFC5424
SD-PARAM-VALUEs instead of rejecting them with a parse error.

sdata-parser(): this new parser allows you to parse an RFC5424 style
structured data string. It can be used to parse this relatively complex
format separately.
(#4281)
system() source: the system() source was changed on systemd platforms to
fetch journal messages that relate to the current boot only (e.g. similar
to journalctl -fb) and to ignore messages generated in previous boots,
even if those messages were succesfully stored in the journal and were not
picked up by syslog-ng. This change was implemented as the journald access
APIs work incorrectly if time goes backwards across reboots, which is an
increasingly frequent event in virtualized environments and on systems that
lack an RTC. If you want to retain the old behaviour, please bypass the
system() source and use systemd-journal() directly, where this option
can be customized. The change is not tied to @version as we deemed the new
behaviour fixing an actual bug. For more information consult #2836.

systemd-journald() source: add match-boot() and matches() options to
allow you to constrain the collection of journal records to a subset of what
is in the journal. match-boot() is a yes/no value that allows you to fetch
messages that only relate to the current boot. matches() allows you to
specify one or more filters on journal fields.

Examples:
```
source s_journal_current_boot_only {
  systemd-source(match-boot(yes));
};

source s_journal_systemd_only {
  systemd-source(matches(
    "_COMM" => "systemd"
    )
  );
};
```
(#4245)
date-parser(): add value() parameter to instruct date-parser() to store
the resulting timestamp in a name-value pair, instead of changing the
timestamp value of the LogMessage.

datetime type representation: typed values in syslog-ng are represented as
strings when stored as a part of a log message. syslog-ng simply remembers
the type it was stored as. Whenever the value is used as a specific type in
a type-aware context where we need the value of the specific type, an
automatic string parsing takes place. This parsing happens for instance
whenever syslog-ng stores a datetime value in MongoDB or when
$(format-date) template function takes a name-value pair as parameter.
The datetime() type has stored its value as the number of milliseconds since
the epoch (1970-01-01 00:00:00 GMT). This has now been enhanced by making
it possible to store timestamps up to nanosecond resolutions along with an
optional timezone offset.

$(format-date): when applied to name-value pairs with the datetime type,
use the timezone offset if one is available.
(#4319)
stats: Added syslog-stats() global stats() group option.

E.g.:
```
options {
  stats(
    syslog-stats(no);
  );
};
```
It changes the behavior of counting messages based on different syslog-proto fields,
like SEVERITY, FACILITY, HOST, etc...

Possible values are:
- yes => force enable
- no => force disable
- auto => let stats(level()) decide (old behavior)
  (#4337)
kubernetes source: Added key-delimiter() option.

Some metadata fields can contain .-s in their name. This does not work
with syslog-ng-s macros, which by default use . as a delimiter. The added
key-delimiter() option changes this behavior by storing the parsed
metadata fields with a custom delimiter. In order to reach the fields, the
accessor side has to use the new delimiter format, e.g. --key-delimiter
option in $(format-json).
(#4213)

Bugfixes

Fix conditional evaluation with a dangling filter

We've fixed a bug that caused conditional evaluation (if/else/elif) and certain logpath flags (final, fallback)
to occasionally malfunction. The issue only happened in certain logpath constructs; examples can be found in the
PR description.
(#4058)
python: Fixed a bug, where PYTHONPATH was ignored with python3.11.
(#4298)
disk-buffer: Fixed disk-queue file becoming corrupt when changing disk-buf-size().

syslog-ng now continues with the originally set disk-buf-size().
Note that changing the disk-buf-size() of an existing disk-queue was never supported,
but could cause errors, which are fixed now.
(#4308)
dqtool: fix dqtool assign
(#4355)
example-diskq-source: Fixed failing to read the disk-queue content in some cases.
([#4308](ht...

Assets 3

21 Dec 20:58

kira-syslogng

syslog-ng-4.0.1

cdc3701

syslog-ng-4.0.1

4.0.1

This is the combination of the news entries of 4.0.0 and 4.0.1.

This is a new major version of syslog-ng, ending the 3.x series which
started roughly 13 years ago, on 17th February 2009.

Like all releases in the 3.x series, 4.0.0 is not a breaking change either.
Long-term compatibility has been and continues to be an essential objective
of syslog-ng; thus, you can still run unchanged configurations that were
originally created for syslog-ng 3.0.0.

You can safely upgrade to 4.0.0 if you followed along 3.x, and you should
probably also consider upgrading if you are stuck with an older 3.x release.

The new version number primarily indicates that this version of syslog-ng is
much more than the software we released 13 years ago. While it does have
certain "big-bang" items in its feature list, new features were continuously
introduced throughout our 3.x series as well. Our engineering practices
have not changed simply because we were working on a new major release: this
is the continuation of our previous releases in every respect, produced in
the same manner, just with a more catchy version number.

For this reason, there is no separate deprecation or support period for 3.x
releases, similarly with our existing practice. We support earlier syslog-ng
releases by providing maintenance and fixes in the new release track.
Fixes to problems are not backported to earlier releases by the syslog-ng
project.

Highlights

Introduce runtime type information to name-value pairs

syslog-ng uses a data model where a log message contains an unordered set
of name-value pairs. The values stored in these name-value pairs are
usually textual, so syslog-ng has traditionally stored these values in
text format.

With the increase of JSON-based message sources and destinations, types
became more important. If we encounter a message where a name-value pair
originates from a JSON document, and this document contains a member that
is numeric, we may want to reproduce that as we send this data to a
consumer.

For example, sometimes we extract a numerical metric from a log message,
and we need to send this to a consumer, again with the correct type.

To be able to do this, we added runtime type information to the syslog-ng
message model: each name-value pair becomes a (name, type, value) triplet.

We introduced the following types:

string: simple textual data, mostly utf8 (but not always)
int: an integer representable by a 64 bit signed value
double: a double precision floating point number
boolean: true or false
datetime: Date and Time represented by the milliseconds since epoch
list: list of strings
json: JSON snippet
null: an unset value

Apart from the syslog-ng core supporting the notion of types, its use is
up to the sources, filters, rewrite rules, parsers and destinations that
set or make use of them in any way it makes the most sense for the component
in question.

Type-aware comparisons

syslog-ng uses filter expressions to make routing decisions and during the
transformation of messages. These filter expressions are used in filter
{} or if {} statements, for example.

In these expressions, you can use comparison operators. This example, for
instance, uses the '>' operator to check for HTTP response codes
greater-or-equal than 500:

     if ("${apache.response}" >= 500) {
     };

Earlier, we had two sets of operators, one for numeric (==, !=, <, >) and the
other for string-based comparisons (eq, ne, gt, lt).

The separate operators were cumbersome to use. Users often forgot which
operator was the right one for a specific case.

Typing allows us to do the right thing in most cases automatically, and a
syntax that allows the user to override the automatic decisions in the
rare case.

With that, starting with 4.0, the old-numeric operators have been
converted to be type-aware operators. It would compare as strings if both
sides of the comparisons are strings. It would compare numerically if at
least one side is numeric. A great deal of inspiration was taken from
JavaScript, which was considered to be a good model, since the problem
space is similar.

See this blog post for more details:
https://syslog-ng-future.blog/syslog-ng-4-progress-3-38-1-release/

Capture type information from JSON

When using json-parser(), syslog-ng converts all members of a JSON object
to syslog-ng name-value pairs. Prior to the introduction of type support,
these name-value pairs were all stored as strings. Any type information
originally present in the incoming JSON object was lost.

This meant that if you regenerated the JSON from the name-value pairs using
the $(format-json) template function, all numbers, booleans and other
types became strings in the output.

There has been a feature in syslog-ng that alleviated the loss of types.
This feature was called "type-hints". Type-hints tell $(format-json) to
use a specific type on output, independently of a name-value pair's
original type, but this type conversion needed to be explicit in the
configuration.

An example configuration that parses JSON on input and produces a JSON on
output:

log {
    source { ... };
    parser { json-parser(prefix('.json.')); };
    destination { file(... template("$(format-json .json.*)\n")); };
};

To augment the above with type hinting, you could use:

log {
    source { ... };
    parser { json-parser(prefix('.json.')); };
    destination { file(... template("$(format-json .json.* .json.value=int64(${.json.value})\n")); };
};

NOTE the presence of the int64() type hint in the 2nd example.

The new feature introduced with typing is that syslog-ng would
automatically store the JSON type information as a syslog-ng type, thus it
will transparently carry over types from inputs to output, without having
to be explicit about them.

Typing support for various components in syslog-ng

Typing is a feature throughout syslog-ng, and although the gust of it has
been explained in the highlights section, some further details are
documented in the list down below:

type-aware comparisons in filter expressions: as detailed above, the
previously numeric operators become type-aware, and the exact comparison
performed will be based on types associated with the values we compare.
json-parser() and $(format-json): JSON support is massively improved
with the introduction of types. For one: type information is retained
across input parsing->transformation->output formatting. JSON lists
(arrays) are now supported and are converted to syslog-ng lists so they
can be manipulated using the $(list-*) template functions. There are
other important improvements in how we support JSON.
set(), groupset(): in any case where we allow the use of templates,
support for type-casting was added, and the type information is properly
promoted.
db-parser() type support: db-parser() gets support for type casts,
assignments within db-parser() rules can associate types with
values using the "type" attribute, e.g. <value name="foobar" type="integer">$PID</value>. The “integer” is a type-cast that
associates $foobar with an integer type. db-parser()’s internal parsers
(e.g. @NUMBER@) will also associate type information with a name-value
pair automatically.
add-contextual-data() type support: any new name-value pair that is
populated using add-contextual-data() will propagate type information,
similarly to db-parser().
map-value-pairs() type support: propagate type information
SQL type support: the sql() driver gained support for types, so that
columns with specific types will be stored as those types.
template type support: templates can now be casted explicitly to a
specific type, but they also propagate type information from
macros/template functions and values in the template string
value-pairs type support: value-pairs form the backbone of specifying a
set of name-value pairs and associated transformations to generate JSON
or a key-value pair format. It also gained support for types, the
existing type-hinting feature that was already part of value-pairs was
adapted and expanded to other parts of syslog-ng.
python() typing: support for typing was added to all Python components
(sources, destinations, parsers and template functions), along with more
documentation & examples on how the Python bindings work. All types except
json() are supported as they are queried- or changed by Python code.
on-disk serialized formats (e.g. disk buffer/logstore): we remain
compatible with messages serialized with an earlier version of
syslog-ng, and the format we choose remains compatible for “downgrades”
as well. E.g. even if a new version of syslog-ng serialized a message,
the old syslog-ng and associated tools will be able to read it (sans
type information of course)

Improved support for lists (arrays)

For syslog-ng, everything is traditionally a string. A convention was
started with syslog-ng in v3.10, where a comma-separated format
could be used as a kind of array using the $(list-*) family of template
functions.

For example, $(list-head) takes off the first element in a list, while
$(list-tail) takes the last. You can index and slice list elements using
the $(list-slice) and $(list-nth) functions and so on.

syslog-ng has started to return such lists in various cases, so they can
be manipulated using these list-specific template functions. These
include the xml-parser(), or the $(explode) template function, but there
are others.

Here is an example that has worked since syslog-ng 3.10:

`...

Assets 3

20 Dec 19:01

kira-syslogng

syslog-ng-4.0.0

49c66d7

syslog-ng-4.0.0

4.0.0

This is a new major version of syslog-ng, ending the 3.x series which
started roughly 13 years ago, on 17th February 2009.

You can safely upgrade to 4.0.0 if you followed along 3.x, and you should
probably also consider upgrading if you are stuck with an older 3.x release.

Highlights

Introduce runtime type information to name-value pairs

For example, sometimes we extract a numerical metric from a log message,
and we need to send this to a consumer, again with the correct type.

To be able to do this, we added runtime type information to the syslog-ng
message model: each name-value pair becomes a (name, type, value) triplet.

We introduced the following types:

string: simple textual data, mostly utf8 (but not always)
int: an integer representable by a 64 bit signed value
double: a double precision floating point number
boolean: true or false
datetime: Date and Time represented by the milliseconds since epoch
list: list of strings
json: JSON snippet
null: an unset value

Type-aware comparisons

syslog-ng uses filter expressions to make routing decisions and during the
transformation of messages. These filter expressions are used in filter
{} or if {} statements, for example.

In these expressions, you can use comparison operators. This example, for
instance, uses the '>' operator to check for HTTP response codes
greater-or-equal than 500:

     if ("${apache.response}" >= 500) {
     };

Earlier, we had two sets of operators, one for numeric (==, !=, <, >) and the
other for string-based comparisons (eq, ne, gt, lt).

The separate operators were cumbersome to use. Users often forgot which
operator was the right one for a specific case.

Typing allows us to do the right thing in most cases automatically, and a
syntax that allows the user to override the automatic decisions in the
rare case.

See this blog post for more details:
https://syslog-ng-future.blog/syslog-ng-4-progress-3-38-1-release/

Capture type information from JSON

This meant that if you regenerated the JSON from the name-value pairs using
the $(format-json) template function, all numbers, booleans and other
types became strings in the output.

An example configuration that parses JSON on input and produces a JSON on
output:

log {
    source { ... };
    parser { json-parser(prefix('.json.')); };
    destination { file(... template("$(format-json .json.*)\n")); };
};

To augment the above with type hinting, you could use:

log {
    source { ... };
    parser { json-parser(prefix('.json.')); };
    destination { file(... template("$(format-json .json.* .json.value=int64(${.json.value})\n")); };
};

NOTE the presence of the int64() type hint in the 2nd example.

Typing support for various components in syslog-ng

Typing is a feature throughout syslog-ng, and although the gust of it has
been explained in the highlights section, some further details are
documented in the list down below:

type-aware comparisons in filter expressions: as detailed above, the
previously numeric operators become type-aware, and the exact comparison
performed will be based on types associated with the values we compare.
json-parser() and $(format-json): JSON support is massively improved
with the introduction of types. For one: type information is retained
across input parsing->transformation->output formatting. JSON lists
(arrays) are now supported and are converted to syslog-ng lists so they
can be manipulated using the $(list-*) template functions. There are
other important improvements in how we support JSON.
set(), groupset(): in any case where we allow the use of templates,
support for type-casting was added, and the type information is properly
promoted.
db-parser() type support: db-parser() gets support for type casts,
assignments within db-parser() rules can associate types with
values using the "type" attribute, e.g. <value name="foobar" type="integer">$PID</value>. The “integer” is a type-cast that
associates $foobar with an integer type. db-parser()’s internal parsers
(e.g. @NUMBER@) will also associate type information with a name-value
pair automatically.
add-contextual-data() type support: any new name-value pair that is
populated using add-contextual-data() will propagate type information,
similarly to db-parser().
map-value-pairs() type support: propagate type information
SQL type support: the sql() driver gained support for types, so that
columns with specific types will be stored as those types.
template type support: templates can now be casted explicitly to a
specific type, but they also propagate type information from
macros/template functions and values in the template string
value-pairs type support: value-pairs form the backbone of specifying a
set of name-value pairs and associated transformations to generate JSON
or a key-value pair format. It also gained support for types, the
existing type-hinting feature that was already part of value-pairs was
adapted and expanded to other parts of syslog-ng.
python() typing: support for typing was added to all Python components
(sources, destinations, parsers and template functions), along with more
documentation & examples on how the Python bindings work. All types except
json() are supported as they are queried- or changed by Python code.
on-disk serialized formats (e.g. disk buffer/logstore): we remain
compatible with messages serialized with an earlier version of
syslog-ng, and the format we choose remains compatible for “downgrades”
as well. E.g. even if a new version of syslog-ng serialized a message,
the old syslog-ng and associated tools will be able to read it (sans
type information of course)

Improved support for lists (arrays)

For example, $(list-head) takes off the first element in a list, while
$(list-tail) takes the last. You can index and slice list elements using
the $(list-slice) and $(list-nth) functions and so on.

Here is an example that has worked since syslog-ng 3.10:

  # MSG contains foo:bar:baz
  # - the $(list-head) take...

Assets 3

2 Join discussion

29 Aug 11:11

kira-syslogng

syslog-ng-3.38.1

7a087fa

syslog-ng-3.38.1

3.38.1

Highlights

Sneak peek into syslog-ng v4.0

syslog-ng v4.0 is right around the corner.

This release (v3.38.1) contains all major changes, however, they are
currently all hidden behind a feature flag.
To enable and try those features, you need to specify @version: 4.0 at the
top of the configuration file.

You can find out more about the 4.0 changes and features here.

Read our practical introduction to typing at
syslog-ng-future.blog.

Features

grouping-by(): added inject-mode(aggregate-only)

This inject mode will drop individual messages that make up the correlation
context (key() groups) and would only yield the aggregate messages
(e.g. the results of the correlation).
(#3998)
add-contextual-data(): add support for type propagation, e.g. set the
type of name-value pairs as they are created/updated to the value returned
by the template expression that we use to set the value.

The 3rd column in the CSV file (e.g. the template expression) now supports
specifying a type-hint, in the format of "type-hint(template-expr)".

Example line in the CSV database:

selector-value,name-value-pair-to-be-created,list(foo,bar,baz)
(#4051)
$(format-json): add --key-delimiter option to reconstruct JSON objects
using an alternative structure separator, that was created using the
key-delimiter() option of json-parser().
(#4093)
json-parser(): add key-delimiter() option to extract JSON structure
members into name-value pairs, so that the names are flattened using the
character specified, instead of dot.

Example:
Input: {"foo":{"key":"value"}}

Using json-parser() without key-delimiter() this is extracted to:
```
  foo.key="value"
```
Using json-parser(key-delimiter("~")) this is extracted to:
```
  foo~key="value"
```
This feature is useful in case the JSON keys contain dots themselves, in
those cases the syslog-ng representation is ambigious.
(#4093)

Bugfixes

Fixed buffer handling of syslog and timestamp parsers (CVE-2022-38725)

Multiple buffer out-of-bounds issues have been fixed, which could cause
hangs, high CPU usage, or other undefined behavior.
(#4110)
Fixed building with LibreSSL
(#4081)
network(): Fixed a bug, where syslog-ng halted the input instead of skipping a character
in case of a character conversion error.
(#4084)
redis(): Fixed bug where using redis driver without the batch-lines option caused program crash.
(#4114)
pdbtool: fix a SIGABRT on FreeBSD that was triggered right before pdbtool
exits. Apart from being an ugly crash that produces a core file,
functionally the tool behaved correctly and this case does not affect
syslog-ng itself.
(#4037)
regexp-parser(): due to a change introduced in 3.37, named capture groups
are stored indirectly in the LogMessage to avoid copying of the value. In
this case the name-value pair created with the regexp is only stored as a
reference (name + length of the original value), which improves performance
and makes such name-value pairs use less memory. One omission in the
original change in 3.37 is that syslog-ng does not allow builtin values to
be stored indirectly (e.g. $MESSAGE and a few of others) and this case
causes an assertion to fail and syslog-ng to crash with a SIGABRT. This
abort is now fixed. Here's a sample config that reproduces the issue:
```
regexp-parser(patterns('(?<MESSAGE>.*)'));
```
(#4043)
set-tag: fix cloning issue when string literal were used (see #4062)
(#4065)
add-contextual-data(): fix high memory usage when using large CSV files
(#4067)

Other changes

The json-c library is no longer bundled in the syslog-ng source tarball

Since all known OS package managers provide json-c packages nowadays, the json-c
submodule has been removed from the source tarball.

The --with-jsonc=internal option of the configure script has been removed
accordingly, system libraries will be used instead. For special cases, the JSON
support can be disabled by specifying --with-jsonc=no.
(#4078)
platforms: Dropped support for ubuntu-impish as it became EOL
(#4088)

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Alvin Šipraga, Andras Mitzki, Attila Szakacs, Balazs Scheidler,
Bálint Horváth, Daniel Klauer, Fabrice Fontaine, Gabor Nagy,
HenryTheSir, László Várady, Parrag Szilárd, Peter Kokai, Shikhar Vashistha,
Szilárd Parrag, Vivin Peris

Assets 3

07 Jun 14:41

kira-syslogng

syslog-ng-3.37.1

d78b130

syslog-ng-3.37.1

3.37.1

Highlights

kubernetes source: A new source for Kubernetes CRI (Container Runtime Interface) format.
By default it tails the /var/log/containers folder which can be overriden with the base-dir() parameter.
Example configuration:
```
source {
  kubernetes();
  # or specifying the directory:
  # kubernetes(base-dir("/dir/to/tail"));
};
```
(#4015)
mariadb-audit-parser: A new parser for mariadb/mysql audit plugin logs have been added.
The parser supports the syslog output type's format, see mariadb page for details.
(#3947)

Features

internal(): add rcptid tag to all trace messages that relate to incoming
log messages. This makes it easier to correlate parsing, rewriting and
routing actions with incoming log messages.
(#3972)
syslog-parser(): allow comma (e.g. ',') to separate the seconds and the fraction of a
second part as some devices use that character. This change applies to both
to syslog-parser() and the builtin syslog parsing functionality of network
source drivers (e.g. udp(), tcp(), network() and syslog()).
(#3949)
cisco-parser: add ISO 8601 timestamp support
(#3934)
network(), syslog() sources and destinations: added new TLS options sigalgs() and client-sigalgs()

They can be used to restrict which signature/hash pairs can be used in digital signatures.
It sets the "signature_algorithms" extension specified in RFC5246 and RFC8446.

Example configuration:
```
destination {
    network("test.host" port(4444) transport(tls)
        tls(
            pkcs12-file("/path/to/tls/test.p12")
            peer-verify(yes)
            sigalgs("RSA-PSS+SHA256:ed25519")
        )
    );
};
```
(#4000)
set-matches() and unset-matches(): these new rewrite operations allow
the setting of match variables ($1, $2, ...) in a single operation, based
on a syslog-ng list expression.
Example:
```
# set $1, $2 and $3 respectively
set-matches("foo,bar,baz");

# likewise, but using a list function
set-matches("$(explode ':' 'foo:bar:baz')");
```
(#3948)
$* macro: the $* macro in template expressions convert the match variables
(e.g. $1, $2, ...) into a syslog-ng list that can be further manipulated
using the list template functions, or turned into a list in type-aware
destinations.
(#3948)
set-tag(): add support for using template expressions in set-tag() rewrite
operations, which makes it possible to use tag names that include macro
references.
(#3962)

Bugfixes

http() and other threaded destinations: fix $SEQNUM processing so that
only local messages get an associated $SEQNUM, just like normal
syslog()-like destinations. This avoids a [meta sequenceId="XXX"] SD-PARAM
being added to $SDATA for non-local messages.
(#3928)
grouping-by(): fix grouping-by() use through parser references.
Originally if a grouping-by() was part of a named parser statement and was
referenced from multiple log statements, only the first grouping-by()
instance behaved properly, 2nd and subsequent references were ignoring all
configuration options and have reverted using defaults instead.
(#3957)
db-parser(): similarly to grouping-by(), db-parser() also had issues
propagating some of its options to 2nd and subsequent references of a parser
statement. This includes drop-unmatched(), program-template() and
template() options.
(#3957)
match(), subst() and regexp-parser(): fixed storing of numbered
(e.g. $1,$2, $3 and so on) and named capture groups in regular expressions
in case the input of the regexp is the same as one of the match variables being
stored. In some cases the output of the regexp was clobbered and an invalid
value stored.
(#3948)
fix threaded(no) related crash: if threaded mode is disabled for
asynchronous sources and destinations (all syslog-like drivers such as
tcp/udp/syslog/network qualify), a use-after-free condition can happen due
to a reference counting bug in the non-threaded code path. The
threaded(yes) setting has been the default since 3.6.1 so if you are using
a more recent version, you are most probably unaffected. If you are using
threaded(no) a use-after-free condition happens as the connection closes.
The problem is more likely to surface on 32 bit platforms due to pointer
sizes and struct layouts where this causes a NULL pointer dereference.
(#3997)
set(): make sure that template formatting options (such as time-zone() or
frac-digits()) are propagated to all references of the rewrite rule
containing a set(). Previously the clone() operation used to implement
multiple references missed the template related options while cloning set(),
causing template formatting options to be set differently, depending on
where the set() was referenced from.
(#3962)
csv-parser(): fix flags(strip-whitespace) and null-value handling
for greedy column
(#4028)

Other changes

java()/python() destinations: the $SEQNUM macro (and "seqnum" attribute in
Python) was erroneously for both local and non-local logs, while it should
have had a value only in case of local logs to match RFC5424 behavior
(section 7.3.1). This bug is now fixed, but that means that all non-local
logs will have $SEQNUM set to zero from this version on, e.g. the $SEQNUM
macro would expand to an string, to match the syslog() driver behaviour.
(#3928)
dbld: add support for Fedora 35 in favour of Fedora 33
(#3933)
debian: fix logrotate file not doing the file rotation. (The path and command was invalid.)
(#4031)
OpenSSL: add support for OpenSSL 3.0
(#4012)
The MD4 hash function ($(md4)) is no longer available when compiling syslog-ng with OpenSSL v3.0.
MD4 is now deprecated, it will be removed completely in future versions.
(#4012)

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Andras Mitzki, Attila Szakacs, Balazs Scheidler, Ben Burrows,
Fᴀʙɪᴇɴ Wᴇʀɴʟɪ, Gabor Nagy, László Várady, mohitvaid,
Parrag Szilárd, Peter Kokai, Peter Viskup, Roffild,
Ryan Faircloth, Scott Parlane, Zoltan Pallagi

Assets 3

04 Mar 15:04

github-actions

syslog-ng-3.36.1

f26dc2c

syslog-ng-3.36.1

3.36.1

Highlights

system() source: added basic support for reading macOS system logs

The current implementation processes the output of the original macOS syslogd:
/var/log/system.log.
(#3710)
$(values) and $(names): these new template functions can be used to
query a list of name-value pairs in the current message. The list of name
value pairs queried are specified by a value-pairs expression, just like
with $(format-json).

Examples:

This expression sets the JSON array values to contain the list of SDATA
values, while the JSON array names would contain the associated names, in
the same order.

$(format-json values=list($(values .SDATA.*)) names=list($(names .SDATA.*)))

The resulting name-value pairs are always sorted by their key, regardless of
the argument order.
(#3911)
rename(): added a new rewrite rule, called rename()

Example usage:
```
rewrite {
  rename( "renamed-from" "renamed-to" );
};
```
(#3841)

Features

network() drivers: added TLS keylog support

syslog-ng dumps TLS secrets for a given source/destination, which can be used for
debugging purposes to decrypt data with, for example, Wireshark.

This should be used for debugging purposes only!

Example usage:

source tls_source{
  network(
      port(1234)
      transport("tls"),
      tls(
        key-file("/path/to/server_key.pem"),
        cert-file("/path/to/server_cert.pem"),
        ca-dir("/path/to/ca/")
        keylog-file("/path/to/keylog_file")
      )
  );
};

(#3792)

tls() block: added option for restricting TLS 1.3 ciphers

The network(), syslog(), and the http() modules now support specifying TLS 1.3 cipher suites,

Example usage:
```
network(
  transport("tls")
  tls(
    pkcs12-file("test.p12")
    cipher-suite(
      tls12-and-older("ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256"),
      tls13("TLS_CHACHA20_POLY1305_SHA256:TLS_AES_256_GCM_SHA384")
    )
  )
);
```
tls12-and-older() can be used to specify TLS v1.2-and-older ciphers,
tls13() can be used for TLS v1.3 ciphers only.

Note: The old cipher-suite("list:of:ciphers") option restricts only the TLS v1.2-and-older cipher suite
for backward compatibility.
(#3907)
file() destination: added a new option: symlink-as()

This feature allows one to maintain a persistent symlink to a log file when a
template is used (for example: /var/log/cron -> /var/log/cron.${YEAR}${MONTH}).

Example usage:
```
destination d_file_cron {
  file("/var/log/cron.${YEAR}${MONTH}" symlink-as("/var/log/cron"));
};
```
From a functional perspective, the symlink-as file inherits both
create-dirs and file ownership from its file destination (permissions are not
applicable to symlinks, at least on linux).

The symlink is adjusted at the time a new destination file is opened (in the
example above, if ${YEAR} or ${MONTH} changes).

Although not specific to time macros, that's where the usefulness is. If the
template contains something like ${PROGRAM} or ${HOST}, the configuration wouldn't
necessarily be invalid, but you'd get an ever-changing symlink of dubious
usefulness.
(#3855)
flags(no-rfc3164-fallback): added a new flag to sources that parse
incoming syslog data and operate in RFC5424 mode (e.g. syslog-protocol is
also set). With the new flag the automatic fallback to RFC3164 format
is disabled. In this case if the parsing in RFC5424 fails, the
syslog parser would result in an error message. In the case of
syslog-parser(drop-invalid(yes)), the message would be dropped.
(#3891)
syslog-format: accept ISO timestamps that incorrectly use a space instead of
a 'T' to delimit the date from the time portion. For example, a
"2021-01-01T12:12:12" timestamp is well formed according to RFC5424 (which
uses a subset of ISO8601, see https://datatracker.ietf.org/doc/html/rfc5424#section-6.2.3).
Some systems simply use a space instead of a 'T'. The same format is
accepted for both RFC3164 (e.g. udp(), tcp() and network() sources) and
RFC5424 (e.g. syslog() source).
(#3893)
transport(text-with-nuls): added a new transport mechanism for
the network() driver that allows NUL characters within the message.

Note: syslog-ng does not support embedded NUL characters everywhere, so it is
recommended that you also use flags(no-multi-line) that causes NUL
characters to be replaced by space.
(#3913)

Bugfixes

filter: fixed the not operator in filter expressions (regression in v3.35.1)

Reusing a filter that contains the not operator more than once, or
referencing a complex expression containing not might have caused invalid results
in the previous syslog-ng version (v3.35.1). This has been fixed.
(#3863)
throttle() filter: support negation
(#3863)
disk-buffer(): fixed a crash which could happen in very rare cases, while a corrupted disk-buffer was getting replaced
(#3845)
disk-buffer(): fixed a memory leak issue and inconsistent buffer handling in rare cases
(#3887)
disk-buffer(): fixed underflowing queued stats counter
(#3887)
disk-buffer(): fixed queued stats were not adjusted when a disk-buffer became corrupt
(#3851)
disk-buffer(): fixed a disk-buffer corruption issue

A completely filled and then emptied disk-buffer may have been recognised as corrupt.
(#3874)
amqp(): fixed a minor error reporting problem.
(#3869)
amqp(): syslog-ng now drops messages that are too large to send
(#3869)
amqp(): fixed a crash, which happened with librabbitmq v0.9.0 or v0.10.0, while using the tls() block.
(#3929)
file() source: fixed invalid buffer handling when encoding() is used

A bug has been fixed that - under rare circumstances - could cause message
duplication or partial message loss when non-fixed length or less known
fixed-length encodings are used.
(#3892)
syslog-ng: fixed a SIGSEGV triggered by an incorrectly formatted "CONFIG"
command, received on the syslog-ng control socket. The only known
implementation of the control protocol is syslog-ng-ctl itself, which always
sends a correct command, but anyone with access to the UNIX domain socket
syslog-ng.ctl (root only by default) can trigger a crash.
(#3900)
credit-card-mask(): fixed visa, mastercard and jcb card regex pattern
(#3853)
cisco-parser(): allow a leading dot in the timestamp (not synced clocks)
(#3843)

Notes to developers

plugins: we have made it easier to implement filter plugins

An example can be found under modules/rate-limit-filter.
(#3866)
dev-utils: various fixes for the plugin skeleton generator script
(#3866)

Other changes

The syslog-ng Docker image
is now automatically tagged and pushed to Docker Hub after each release
(#3870)
throttle() filter: renamed to rate-limit()
(#3866)
python: support Python 3.10
(#3865)
java: upgraded from old log4j v1.x line to log4j v2.17.2
(#3861)
(#3927)

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Andras Mitzki, Andrea Biardi, Attila Szakacs, Balazs Scheidler,
Balázs Barkó, Benedek Cserhati, Gabor Nagy, Janos SZIGETVARI,
Laszlo Budai, Laszlo Szemere, László Várady, Mikel Olasagasti Uranga,
Norbert Takacs, Parrag Szilárd, Peter Kokai, Szilárd Parrag,
Zoltan Pallagi, Stanislav Osipov, Yash Mathne

Assets 3

15 Nov 11:49

github-actions

syslog-ng-3.35.1

5874f8b

syslog-ng-3.35.1

3.35.1

syslog-ng OSE APT repository

From now on, Ubuntu and Debian packages will be published with every syslog-ng release in the form of an APT repository.

We, syslog-ng developers, provide these packages and the APT repository "as is" without warranty of any kind,
on a best-effort level.

Currently, syslog-ng packages are released for the following distribution versions (x86-64):

Debian: bullseye, buster, stretch, sid, testing
Ubuntu: Impish, Focal, Bionic, Xenial

For instructions on how to install syslog-ng on Debian/Ubuntu distributions, see the
README.

Highlights

throttle(): added a new filter that allows rate limiting messages based on arbitrary keys in each message.
Note: messages over the rate limit are dropped (just like in any other filter).
```
filter f_throttle {
  throttle(
    template("$HOST")
    rate(5000)
  );
};
```
(#3781)
mqtt(): added a new source that can be used to receive messages using the MQTT protocol.
Supported transports: tcp, ws, ssl, wss

Example config:
```
source {
    mqtt{
        topic("sub1"),
        address("tcp://localhost:4445")
    };
};
```
(#3809)

Features

afsocket: Socket options, such as ip-ttl() or tcp-keepalive-time(), are
traditionally named by their identifier defined in socket(7) and unix(7) man
pages. This was not the case with the pass-unix-credentials() option, which -
unlike other similar options - was also possible to set globally.

A new option called so-passcred() is now introduced, which works similarly
how other socket related options do, which also made possible a nice code
cleanup in the related code sections. Of course the old name remains
supported in compatibility modes.

The PR also implements a new source flag ignore-aux-data, which causes
syslog-ng not to propagate transport-level auxiliary information to log
messages. Auxiliary information includes for example the pid/uid of the
sending process in the case of UNIX based transports, OR the X.509
certificate information in case of SSL/TLS encrypted data streams.

By setting flags(ignore-aux-data) one can improve performance at the cost of
making this information unavailable in the log messages received through
affected sources.
(#3670)
network: add support for PROXY header before TLS payload

This new transport method called proxied-tls-passthrough is capable of detecting the
PROXY header before the TLS payload.
Loggen has been updated with the--proxied-tls-passthrough option for testing purposes.
```
source s_proxied_tls_passthrough{
  network(
    port(1234)
    transport("proxied-tls-passthrough"),
    tls(
      key-file("/path/to/server_key.pem"),
      cert-file("/path/to/server_cert.pem"),
      ca-dir("/path/to/ca/")
    )
  );
};
```
(#3770)
mqtt() destination: added client-id option. It specifies the unique client ID sent to the broker.
(#3809)

Bugfixes

unset(), groupunset(): fix unwanted removal of values on different log paths

Due to a copy-on-write bug, unset() and groupunset() not only removed values
from the appropriate log paths, but from all the others where the same message
went through. This has been fixed.
(#3803)
regexp-parser(): fix storing unnamed capture groups under prefix()
(#3810)
loggen: cannot detect plugins on platforms with non .so shared libs (osx)
(#3832)

Packaging

debian/control: Added libcriterion-dev as a build dependency, where it is available from APT.
(debian-bullseye, debian-testing, debian-sid)
(#3794)
centos-7: kafka and mqtt modules are now packaged.

The following packages are used as dependencies:
- librdkafka-devel from EPEL 7
- paho-c-devel from copr:copr.fedorainfracloud.org:czanik:syslog-ng-githead
  (#3797)
debian: Added bullseye support.
(#3794)
bison: support build with bison 3.8
(#3784)

Notes to developers

dbld: As new distributions use python3 by default it makes sense to explicitly state older platforms which use python2
instead of the other way around, so it is not necessary to add that new platform to the python3 case.
(#3780)
dbld: move dbld image cache from DockerHub to GitHub

In 2021, GitHub introduced the GitHub Packages service. Among other
repositories - it provides a standard Docker registry. DBLD uses
this registry, to avoid unnecessary rebuilding of the images.
(#3782)

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Andras Mitzki, Antal Nemes, Attila Szakacs, Balazs Scheidler,
Balázs Barkó, Benedek Cserhati, Colin Douch, Gabor Nagy, Laszlo Szemere,
László Várady, Norbert Takacs, Parrag Szilárd, Peter Czanik (CzP),
Peter Kokai, Robert Paschedag, Ryan Faircloth, Szilárd Parrag,
Thomas Klausner, Zoltan Pallagi

Assets 3

10 Sep 11:58

github-actions

syslog-ng-3.34.1

4b6b8ae

syslog-ng-3.34.1

3.34.1

Highlights

regexp-parser(): new parser that can parse messages with regular expressions

Example:
```
regexp-parser(
  template("${MESSAGE}")
  prefix(".regexp.")
  patterns("(?<DN>foo)", "(?<DN>ball)")
);
```
regexp-parser() can be used as an intuitive replacement for regexp filters
that had their store-matches flag set in order to save those matches.

(#3702)
redis(): workers() and batching support

The Redis driver now support the workers() option, which specifies the
number of parallel workers, and the batch-lines() option.

This could drastically increase the throughput of the Redis destination driver.

Example:
```
redis(
    host("localhost")
    port(6379)
    command("HINCRBY", "hosts", "$HOST", "1")
    workers(8)
    batch-lines(100)
    batch-timeout(10000)
    log-fifo-size(100000)
);
```
(#3732, #3745)

mqtt(): TLS and WebSocket Secure support

The MQTT destination now supports TLS and WSS.

Example config:

mqtt(
  address("ssl://localhost:8883")
  topic("syslog/$HOST")
  fallback-topic("syslog/fallback")

  tls(
    ca-file("/path/to/ca.crt")
    key-file("/path/to/client.key")
    cert-file("/path/to/client.crt")
    peer-verify(yes)
  )
);

(#3747)

Features

system() source: added support for NetBSD
(#3761)
stats: new statistics counter

The following statistics are now available for the HTTP destination, and
other file and network based sources/destinations:
- msg_size_max/msg_size_avg: Shows the largest/average message size of the given source/destination that has
  been measured so far.
- batch_size_max/batch_size_avg: When batching is enabled, then this shows the
  largest/average batch size of the given source/destination that has been measured so far.
- eps_last_1h, eps_last_24h, eps_since_start: Events per second, measured for the last hour,
  for the last 24 hours, and since syslog-ng startup, respectively.
Notes:
- Message sizes are calculated from the incoming raw message length on the source side, and from the outgoing
  formatted message length on the destination side.
- EPS counters are just approximate values, they are updated every minute.
  (#3753)
mqtt(): username/password authentication

Example config:
```
mqtt(
  address("tcp://localhost:1883")
  topic("syslog/messages")
  username("user")
  password("passwd")
);
```
Note: The password is transmitted in cleartext without using ssl:// or wss://.
(#3747)
mqtt(): new option http-proxy() for specifying HTTP/HTTPS proxy for WebSocket connections
(#3747)
syslog-ng-ctl: new flag for pruning statistics

syslog-ng-ctl stats --remove-orphans can be used to remove "orphaned" statistic counters.
It is useful when, for example, a templated file destination ($YEAR.$MONTH.$DAY) produces a lot of stats,
and one wants to remove those abandoned counters occasionally/conditionally.
(#3760)
disk-buffer(): added a new option to reliable disk-buffer: qout-size().

This option sets the number of messages that are stored in the memory in addition
to storing them on disk. The default value is 1000.

This serves performance purposes and offers the same no-message-loss guarantees as
before.

It can be used to maintain a higher throughput when only a small number of messages
are waiting in the disk-buffer.
(#3754)

Bugfixes

network(), syslog(): fixed network sources on NetBSD

On NetBSD, TCP-based network sources closed their listeners shortly after
startup due to a non-portable TCP keepalive setting. This has been fixed.
(#3751)
disk-buffer(): fixed a very rare case, where the reliable disk-buffer never resumed
after triggering flow-control.
(#3752)
disk-buffer(): fixed a rare memory leak that occurred when mem-buf-length()
or mem-buf-size() was configured incorrectly
(#3750)
redis(): fixed command errors that were not detected and marked as successful delivery
(#3748)

Notes to developers

Light framework: new proxy-related options are supported with loggen:
--proxy-src-ip, --proxy-dst-ip, --proxy-src-port, --proxy-dst-port
(#3766)
log-threaded-dest: descendant drivers from LogThreadedDest no longer inherit
batch-lines() and batch-timeout() automatically. Each driver have to opt-in for
these options with log_threaded_dest_driver_batch_option.

log_threaded_dest_driver_option has been renamed to log_threaded_dest_driver_general_option,
and log_threaded_dest_driver_workers_option have been added similarly to the
batch-related options.
(#3741)

Other changes

disk-buffer(): performance improvements

Based on our measurements, the following can be expected compared to the previous syslog-ng release (v3.33.1):
- non-reliable disk buffer: up to 50% performance gain;
- reliable disk buffer: up to 80% increase in performance.
(#3743, #3746, #3754, #3756, #3757)
disk-buffer(): the default value of the following options has been changed for performance reasons:
- truncate-size-ratio(): from 0.01 to 0.1 (from 1% to 10%)
- qout-size(): from 64 to 1000 (this affects only the non-reliable disk buffer)
  (#3757)
kafka-c(): properties-file() option is removed

Please list librdkafka properties in the config() option in syslog-ng's configuration.
See librdkafka configuration here.
(#3704)

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Andras Mitzki, Attila Szakacs, Balazs Scheidler, Balázs Barkó,
Benedek Cserhati, Fabrice Fontaine, Gabor Nagy, Laszlo Szemere,
LittleFish33, László Várady, Norbert Takacs, Parrag Szilárd,
Peter Czanik, Peter Kokai, Zoltan Pallagi

Assets 3

19 Jul 13:37

github-actions

syslog-ng-3.33.2

8037b60

syslog-ng-3.33.2

3.33.2

Bugfixes

disk-buffer: fixed a bug, which was introduced in 3.33.1, where we
sometimes corrupted the disk-buffer file when it reached full size.
(#3726)

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Andras Mitzki, Antal Nemes, Attila Szakacs, Balázs Barkó,
Balazs Scheidler, Benedek Cserhati, Gabor Nagy, Josef Schlehofer,
Laszlo Budai, Laszlo Szemere, László Várady, Norbert Takacs,
Parrag Szilárd, Peter Kokai, Zoltan Pallagi

Assets 3

Releases: syslog-ng/syslog-ng

syslog-ng-4.1.1

4.1.1

Highlights

PROXY protocol v2 support (#4211)

Metrics revised

Prometheus metric format (#4325)

Classification (metadata-based metrics) (#4318)

Named log paths (path ingress/egress metrics) (#4344)

Features

Bugfixes

syslog-ng-4.1.0

4.1.0

Highlights

PROXY protocol v2 support (#4211)

Metrics revised

Prometheus metric format (#4325)

Classification (metadata-based metrics) (#4318)

Named log paths (path ingress/egress metrics) (#4344)

Features

Bugfixes

syslog-ng-4.0.1

4.0.1

Highlights

Introduce runtime type information to name-value pairs

Type-aware comparisons

Capture type information from JSON

Typing support for various components in syslog-ng

Improved support for lists (arrays)

syslog-ng-4.0.0

4.0.0

Highlights

Introduce runtime type information to name-value pairs

Type-aware comparisons

Capture type information from JSON

Typing support for various components in syslog-ng

Improved support for lists (arrays)

syslog-ng-3.38.1

3.38.1

Highlights

Sneak peek into syslog-ng v4.0

Features

Bugfixes

Other changes

Credits

syslog-ng-3.37.1

3.37.1

Highlights

Features

Bugfixes

Other changes

Credits

syslog-ng-3.36.1

3.36.1

Highlights

Features

Bugfixes

Notes to developers

Other changes

Credits

syslog-ng-3.35.1

3.35.1

syslog-ng OSE APT repository

Highlights

Features

Bugfixes

Packaging

Notes to developers

Credits

syslog-ng-3.34.1

3.34.1

Highlights

Features

Bugfixes

Notes to developers

Other changes

Credits

syslog-ng-3.33.2

3.33.2

Bugfixes