[libbeat] : add processors "decode_duration", "move_fields" #31301

kunsonx · 2022-04-14T10:05:29Z

What does this PR do?

Ability to manipulate log fields.

Give a prefix for all log fields
Decode go style duration string then custom format
Move fields from map into another map
Move fields into a parent field map.

Why is it important?

Give users ability custom log field operations.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

cla-checker-service · 2022-04-14T10:05:33Z

💚 CLA has been signed

elasticmachine · 2022-04-14T10:10:35Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-11-15T14:35:14.716+0000
Duration: 67 min 46 sec

Test stats 🧪

Test	Results
Failed	0
Passed	23951
Skipped	1951
Total	25902

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2022-04-14T15:05:03Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

belimawr

@kunsonx Are you still working on this PR? On a quick glance it looks fine, but it's missing tests and documentation.

Could you request a re-review once you're done with coding?

kunsonx · 2022-04-22T09:26:30Z

@kunsonx Are you still working on this PR? On a quick glance it looks fine, but it's missing tests and documentation.

Could you request a re-review once you're done with coding?

OK. I will finish this works. Should I close this MR and open again when it's done?

@belimawr

BTW Where is the guideline for write tests and documentation?

I am beginner for this project contribute.

belimawr · 2022-04-25T16:21:24Z

@kunsonx Are you still working on this PR? On a quick glance it looks fine, but it's missing tests and documentation.
Could you request a re-review once you're done with coding?

OK. I will finish this works. Should I close this MR and open again when it's done?

@belimawr

BTW Where is the guideline for write tests and documentation?

I am beginner for this project contribute.

No need to close/open another PR, just make sure to use the re-review request button once you're done so I get notified and can start reviewing.

You can check the contributing docs here: https://www.elastic.co/guide/en/beats/devguide/current/beats-contributing.html, there you will also find information regarding tests and documentation.

If you still need something more specific, then try looking how the other processors implement their tests and documentation.

kunsonx · 2022-05-11T09:32:51Z

@belimawr Hi. I finished the add_prefix processor tests and documentation. Could you take a look it is satisfaction? if you feel it is ok then will finish these other processors.

Thanks for your time.

belimawr · 2022-05-12T10:20:45Z

@belimawr Hi. I finished the add_prefix processor tests and documentation. Could you take a look it is satisfaction? if you feel it is ok then will finish these other processors.

Thanks for your time.

Hey @kunsonx I can already see some linters are failing, could you take a look at them? They're quite strict, so don't be dismayed by it. Usually it takes some iterations until all linter passes.

To run it locally, go the root of the repo and run

mage linter:lastChange

You will have to fix all lint issues in the files you changed (not only the lines you added).

Because we added this linter recently, sometimes we have to ask the linter to ignore some lines/functions with a //nolint directive.

belimawr · 2022-06-28T15:29:38Z

I'm closing it because there hasn't been any updated in a while.

@kunsonx, if you still want to work on this, just re-open the PR and push the new commits ;)

kunsonx · 2022-09-19T11:26:05Z

@belimawr Hi belimawr.

Just busy for while and rework of these processors.

Just take a look it is satisfy?

#33119

Thanks.

mergify · 2022-09-19T16:45:43Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @kunsonx? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

kunsonx · 2022-09-25T16:00:20Z

Features sync from #33119. please take a look.

Thanks. @belimawr

belimawr

Great Job!

I added some comments regarding the implementation and some documentation copy. I'll also try to get another reviewer at least for the docs.

libbeat/processors/decode_duration/decode_duration.go

libbeat/processors/decode_duration/decode_duration_test.go

libbeat/processors/decode_duration/docs/decode_duration.asciidoc

libbeat/processors/decode_duration/decode_duration.go

libbeat/processors/move_fields/move_fields.go

libbeat/processors/move_fields/move_fields_test.go

libbeat/processors/decode_duration/decode_duration_test.go

kunsonx · 2022-10-19T08:35:57Z

@dedemorton Thanks for your suggestions it make the docs better. Still has two suggestion here need more helps. 😀

belimawr · 2022-11-01T09:10:02Z

@dedemorton could you take another look/help out here?

kunsonx · 2022-11-14T03:17:32Z

@dedemorton Have any idea about the pull request?

dedemorton

The changes LGTM!

belimawr · 2022-11-15T14:32:04Z

@kunsonx could you update the PR with main? You can use the Update branch button here or do it manually.

There were some big changes in the codebase recently, but they should not affect this PR.

Co-authored-by: DeDe Morton <[email protected]>

kunsonx · 2022-11-15T14:35:41Z

@kunsonx could you update the PR with main? You can use the Update branch button here or do it manually.

There were some big changes in the codebase recently, but they should not affect this PR.

Done.

kunsonx · 2022-11-16T03:58:46Z

@belimawr Seems had two reviewers approved this pull request. so what is next?

dedemorton · 2022-11-16T18:53:19Z

I've added the backport label to make sure this gets into the next release. It sounds like this is ready to merge!

Co-authored-by: hxms <[email protected]> Co-authored-by: DeDe Morton <[email protected]> (cherry picked from commit dc1a0ca)

belimawr · 2022-11-16T19:11:58Z

@belimawr Seems had two reviewers approved this pull request. so what is next?

Merge and wait for it to land in the next release ;)

Thanks a lot for the contribution!

belimawr · 2022-11-16T19:15:31Z

I was reviewing the backport and realised we missed the changelog entry 🤦‍♂️

@kunsonx would you mind creating a new PR adding the changelog entry? You can tag me to review it.

kunsonx · 2022-11-21T09:55:23Z

The changelog update at pull request: #33714
@belimawr

…3700) Co-authored-by: hxms <[email protected]> Co-authored-by: DeDe Morton <[email protected]> (cherry picked from commit dc1a0ca) Co-authored-by: 冰天雪地 <[email protected]> Co-authored-by: Denis <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

Co-authored-by: hxms <[email protected]> Co-authored-by: DeDe Morton <[email protected]>

kunsonx requested a review from a team as a code owner April 14, 2022 10:05

kunsonx requested review from belimawr and fearful-symmetry and removed request for a team April 14, 2022 10:05

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Apr 14, 2022

mergify bot assigned kunsonx Apr 14, 2022

kunsonx changed the title ~~feat : add libbeat processors "add_prefix", "decode_duration", "move_fields", "objectize".~~ [libbeat] : add processors "add_prefix", "decode_duration", "move_fields", "objectize". Apr 14, 2022

belimawr added the Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team label Apr 14, 2022

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Apr 14, 2022

belimawr reviewed Apr 20, 2022

View reviewed changes

kunsonx requested a review from belimawr May 11, 2022 09:28

belimawr closed this Jun 28, 2022

kunsonx changed the title ~~[libbeat] : add processors "add_prefix", "decode_duration", "move_fields", "objectize".~~ [libbeat] : add processors "decode_duration", "move_fields" Sep 19, 2022

kunsonx mentioned this pull request Sep 19, 2022

[libbeat] : add processors "decode_duration", "move_fields" #33119

Closed

6 tasks

belimawr reopened this Sep 19, 2022

kunsonx force-pushed the main branch from 5cf2715 to 80e5aed Compare September 25, 2022 15:58

belimawr requested changes Oct 10, 2022

View reviewed changes

kunsonx removed the request for review from fearful-symmetry October 10, 2022 18:07

kunsonx requested review from dedemorton and removed request for belimawr and dedemorton October 19, 2022 08:36

dedemorton approved these changes Nov 14, 2022

View reviewed changes

kunsonx requested a review from belimawr November 15, 2022 03:16

belimawr approved these changes Nov 15, 2022

View reviewed changes

hxms and others added 4 commits November 15, 2022 22:34

feat : add libbeat processor decode_duration/move_fields.

47c6b59

Apply suggestions from code review

d7e9a35

Co-authored-by: DeDe Morton <[email protected]>

feat : add sub test to processor "move_fields"

e2ced63

Apply suggestions from code review

8a17e9c

Co-authored-by: DeDe Morton <[email protected]>

kunsonx force-pushed the main branch from 74f5270 to 8a17e9c Compare November 15, 2022 14:34

dedemorton added the backport-v8.6.0 Automated backport with mergify label Nov 16, 2022

belimawr merged commit dc1a0ca into elastic:main Nov 16, 2022

mergify bot pushed a commit that referenced this pull request Nov 16, 2022

libbeat: add processors "decode_duration", "move_fields" (#31301)

b0fff93

Co-authored-by: hxms <[email protected]> Co-authored-by: DeDe Morton <[email protected]> (cherry picked from commit dc1a0ca)

mergify bot mentioned this pull request Nov 16, 2022

[8.6](backport #31301) [libbeat] : add processors "decode_duration", "move_fields" #33700

Merged

kunsonx mentioned this pull request Nov 17, 2022

[changelogs] : add processors "decode_duration", "move_fields" #33714

Merged

dedemorton mentioned this pull request Jan 25, 2023

[REQUEST]: Add decode_duration processor docs to Fleet/Agent Guide elastic/observability-docs#2545

Closed

chrisberkhout pushed a commit that referenced this pull request Jun 1, 2023

libbeat: add processors "decode_duration", "move_fields" (#31301)

2ffae42

Co-authored-by: hxms <[email protected]> Co-authored-by: DeDe Morton <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[libbeat] : add processors "decode_duration", "move_fields" #31301

[libbeat] : add processors "decode_duration", "move_fields" #31301

kunsonx commented Apr 14, 2022 •

edited

Loading

cla-checker-service bot commented Apr 14, 2022 •

edited

Loading

elasticmachine commented Apr 14, 2022 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

elasticmachine commented Apr 14, 2022

belimawr left a comment

kunsonx commented Apr 22, 2022 •

edited

Loading

belimawr commented Apr 25, 2022

kunsonx commented May 11, 2022

belimawr commented May 12, 2022

belimawr commented Jun 28, 2022

kunsonx commented Sep 19, 2022

mergify bot commented Sep 19, 2022

kunsonx commented Sep 25, 2022

belimawr left a comment

kunsonx commented Oct 19, 2022

belimawr commented Nov 1, 2022

kunsonx commented Nov 14, 2022

dedemorton left a comment

belimawr commented Nov 15, 2022

kunsonx commented Nov 15, 2022

kunsonx commented Nov 16, 2022

dedemorton commented Nov 16, 2022

belimawr commented Nov 16, 2022

belimawr commented Nov 16, 2022

kunsonx commented Nov 21, 2022

[libbeat] : add processors "decode_duration", "move_fields" #31301

[libbeat] : add processors "decode_duration", "move_fields" #31301

Conversation

kunsonx commented Apr 14, 2022 • edited Loading

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

cla-checker-service bot commented Apr 14, 2022 • edited Loading

elasticmachine commented Apr 14, 2022 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

elasticmachine commented Apr 14, 2022

belimawr left a comment

Choose a reason for hiding this comment

kunsonx commented Apr 22, 2022 • edited Loading

belimawr commented Apr 25, 2022

kunsonx commented May 11, 2022

belimawr commented May 12, 2022

belimawr commented Jun 28, 2022

kunsonx commented Sep 19, 2022

mergify bot commented Sep 19, 2022

kunsonx commented Sep 25, 2022

belimawr left a comment

Choose a reason for hiding this comment

kunsonx commented Oct 19, 2022

belimawr commented Nov 1, 2022

kunsonx commented Nov 14, 2022

dedemorton left a comment

Choose a reason for hiding this comment

belimawr commented Nov 15, 2022

kunsonx commented Nov 15, 2022

kunsonx commented Nov 16, 2022

dedemorton commented Nov 16, 2022

belimawr commented Nov 16, 2022

belimawr commented Nov 16, 2022

kunsonx commented Nov 21, 2022

kunsonx commented Apr 14, 2022 •

edited

Loading

cla-checker-service bot commented Apr 14, 2022 •

edited

Loading

elasticmachine commented Apr 14, 2022 •

edited by jenkins-beats-ci bot

Loading

kunsonx commented Apr 22, 2022 •

edited

Loading