Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Type support add-contextual-data (4.0) #4051

Merged
merged 4 commits into from
Jul 8, 2022
Merged

Type support add-contextual-data (4.0) #4051

merged 4 commits into from
Jul 8, 2022

Conversation

bazsi
Copy link
Collaborator

@bazsi bazsi commented Jun 21, 2022

This PR adds type support to add-contextual-data(), two things:

  • the ability to propagate the type-hint from LogTemplate to a newly set name-value pair
  • the ability to use type-hints in the add-contextual-data() database, in the format of "type-hint(template_expression)"

@kira-syslogng
Copy link
Contributor

Build SUCCESS

bazsi added 4 commits June 23, 2022 09:16
@bazsi
templates: extract log_template_compile_with_type_hint() function fro…
653d3a1 
…m value-pairs

This function takes a "type(template-expr)" formatted string and compiles
it into a template with a type hint, just as we did that in
value-pairs/cmdline.c

Signed-off-by: Balazs Scheidler <[email protected]>
@bazsi
add-contextual-data: add support for type hints in the CSV database
005c94f 
Signed-off-by: Balazs Scheidler <[email protected]>
@bazsi
add-contextual-data: store type information returned by the template
7e310f7 
Signed-off-by: Balazs Scheidler <[email protected]>
@bazsi
news: add news entry
056b509 
Signed-off-by: Balazs Scheidler <[email protected]>
@bazsi bazsi force-pushed the type-support-add-contextual-data branch from 13c0127 to 056b509 Compare June 23, 2022 07:16
@kira-syslogng
Copy link
Contributor

Build SUCCESS

@MrAnno MrAnno added this to the syslog-ng-4.0 milestone Jun 23, 2022
@MrAnno MrAnno self-requested a review July 5, 2022 14:23
@OverOrion OverOrion self-requested a review July 7, 2022 14:18
OverOrion
OverOrion approved these changes Jul 8, 2022
View reviewed changes
Copy link
Collaborator

@OverOrion OverOrion left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@OverOrion OverOrion merged commit 9a37c9e into syslog-ng:master Jul 8, 2022
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this pull request Dec 15, 2022
@0-wiz-0
syslog-ng*: update to 3.38.1
ed7e52d 
snmp module untested since net-snmp doesn't currently build on NetBSD-current

Changes:

3.38.1
======

## Highlights

### Sneak peek into syslog-ng v4.0

syslog-ng v4.0 is right around the corner.

This release (v3.38.1) contains all major changes, however, they are
currently all hidden behind a feature flag.
To enable and try those features, you need to specify `@version: 4.0` at the
top of the configuration file.

You can find out more about the 4.0 changes and features
[here](https://github.com/syslog-ng/syslog-ng/blob/master/NEWS-4.0.md).

Read our practical introduction to typing at
[syslog-ng-future.blog](https://syslog-ng-future.blog/syslog-ng-4-progress-3-38-1-release/).

## Features

  * `grouping-by()`: added `inject-mode(aggregate-only)`

    This inject mode will drop individual messages that make up the correlation
    context (`key()` groups) and would only yield the aggregate messages
    (e.g. the results of the correlation).
    ([#3998](syslog-ng/syslog-ng#3998))
  * `add-contextual-data()`: add support for type propagation, e.g. set the
    type of name-value pairs as they are created/updated to the value returned
    by the template expression that we use to set the value.

    The 3rd column in the CSV file (e.g. the template expression) now supports
    specifying a type-hint, in the format of "type-hint(template-expr)".

    Example line in the CSV database:

    selector-value,name-value-pair-to-be-created,list(foo,bar,baz)
    ([#4051](syslog-ng/syslog-ng#4051))
  * `$(format-json)`: add --key-delimiter option to reconstruct JSON objects
    using an alternative structure separator, that was created using the
    key-delimiter() option of json-parser().
    ([#4093](syslog-ng/syslog-ng#4093))
  * `json-parser()`: add key-delimiter() option to extract JSON structure
    members into name-value pairs, so that the names are flattened using the
    character specified, instead of dot.

    Example:
      Input: {"foo":{"key":"value"}}

      Using json-parser() without key-delimiter() this is extracted to:

          foo.key="value"

      Using json-parser(key-delimiter("~")) this is extracted to:

          foo~key="value"

    This feature is useful in case the JSON keys contain dots themselves, in
    those cases the syslog-ng representation is ambigious.
    ([#4093](syslog-ng/syslog-ng#4093))

## Bugfixes

  * Fixed buffer handling of syslog and timestamp parsers

    Multiple buffer out-of-bounds issues have been fixed, which could cause
    hangs, high CPU usage, or other undefined behavior.
    ([#4110](syslog-ng/syslog-ng#4110))
  * Fixed building with LibreSSL
    ([#4081](syslog-ng/syslog-ng#4081))
  * `network()`: Fixed a bug, where syslog-ng halted the input instead of skipping a character
    in case of a character conversion error.
    ([#4084](syslog-ng/syslog-ng#4084))
  * `redis()`: Fixed bug where using redis driver without the `batch-lines` option caused program crash.
    ([#4114](syslog-ng/syslog-ng#4114))
  * `pdbtool`: fix a SIGABRT on FreeBSD that was triggered right before pdbtool
    exits. Apart from being an ugly crash that produces a core file,
    functionally the tool behaved correctly and this case does not affect
    syslog-ng itself.
    ([#4037](syslog-ng/syslog-ng#4037))
  * `regexp-parser()`: due to a change introduced in 3.37, named capture groups
    are stored indirectly in the LogMessage to avoid copying of the value.  In
    this case the name-value pair created with the regexp is only stored as a
    reference (name + length of the original value), which improves performance
    and makes such name-value pairs use less memory.  One omission in the
    original change in 3.37 is that syslog-ng does not allow builtin values to
    be stored indirectly (e.g.  $MESSAGE and a few of others) and this case
    causes an assertion to fail and syslog-ng to crash with a SIGABRT. This
    abort is now fixed. Here's a sample config that reproduces the issue:

        regexp-parser(patterns('(?<MESSAGE>.*)'));
    ([#4043](syslog-ng/syslog-ng#4043))
  * set-tag: fix cloning issue when string literal were used (see #4062)
    ([#4065](syslog-ng/syslog-ng#4065))
  * `add-contextual-data()`: fix high memory usage when using large CSV files
    ([#4067](syslog-ng/syslog-ng#4067))

## Other changes

  * The `json-c` library is no longer bundled in the syslog-ng source tarball

    Since all known OS package managers provide json-c packages nowadays, the json-c
    submodule has been removed from the source tarball.

    The `--with-jsonc=internal` option of the `configure` script has been removed
    accordingly, system libraries will be used instead. For special cases, the JSON
    support can be disabled by specifying `--with-jsonc=no`.
    ([#4078](syslog-ng/syslog-ng#4078))
  * platforms: Dropped support for ubuntu-impish as it became EOL
    ([#4088](syslog-ng/syslog-ng#4088))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@OverOrion OverOrion OverOrion approved these changes

@MrAnno MrAnno MrAnno approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
syslog-ng-4.0
Development

Successfully merging this pull request may close these issues.
4 participants
@bazsi @kira-syslogng @MrAnno @OverOrion