[Zeek] Add additional data sets #3340

legoguy1000 · 2022-05-11T15:46:01Z

What does this PR do?

Add new data sets for known_hosts, known_certs, known_services, & software logs files.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Relates Additional zeek module log files #3288

Screenshots

elasticmachine · 2022-05-11T15:49:42Z

❕ Build Aborted

The PR is not allowed to run in the CI yet

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-05-11T15:47:16.897+0000
Duration: 2 min 23 sec

Steps errors

Expand to view the steps failures

`Google Storage Download`

Took 0 min 0 sec . View more details here

`Error signal`

Took 0 min 0 sec . View more details here
Description: githubApiCall: The REST API call https://api.github.com/orgs/elastic/members/legoguy1000 return the message : java.lang.Exception: httpRequest: Failure connecting to the service https://api.github.com/orgs/elastic/members/legoguy1000 : httpRequest: Failure connecting to the service https://api.github.com/orgs/elastic/members/legoguy1000 : Code: 404Error: {"message":"User does not exist or is not a member of the organization","documentation_url":"https://docs.github.com/rest/reference/orgs#check-organization-membership-for-a-user"}

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2022-05-12T07:58:43Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-06-28T01:49:12.499+0000
Duration: 71 min 30 sec

Test stats 🧪

Test	Results
Failed	0
Passed	219
Skipped	0
Total	219

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2022-05-12T15:00:46Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

packages/zeek/_dev/build/docs/README.md

packages/zeek/data_stream/software/fields/ecs.yml

packages/zeek/data_stream/software/fields/fields.yml

packages/zeek/data_stream/known_certs/elasticsearch/ingest_pipeline/default.yml

packages/zeek/_dev/build/docs/README.md

packages/zeek/data_stream/known_services/fields/ecs.yml

packages/zeek/_dev/build/docs/README.md

efd6 · 2022-05-16T01:11:37Z

/test

legoguy1000 · 2022-05-20T02:29:24Z

@efd6 can we retest

efd6 · 2022-05-20T03:33:07Z

/test

elasticmachine · 2022-05-20T04:15:52Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`43/43`)	💚
Files	96.341% (`79/82`)	👎 -0.218
Classes	96.341% (`79/82`)	👎 -0.218
Methods	86.267% (`647/750`)	👎 -2.856
Lines	92.968% (`8620/9272`)	👍 2.933
Conditionals	100.0% (`0/0`)	💚

efd6 · 2022-05-20T05:00:04Z

/test

legoguy1000 · 2022-05-27T23:46:27Z

fixed the test error. Ready to retest.

efd6 · 2022-06-14T01:24:17Z

/test

legoguy1000 · 2022-06-14T23:59:48Z

@efd6 Any idea about the error from Elastic Machine or from Jenkins found 0 hits in logs-zeek.known_certs-ep data stream: index_not_found_exception: no such index [logs-zeek.known_certs-ep] Status=404??

efd6

Incidental to looking into the issue:

--- Test results for package: zeek - START ---
FAILURE DETAILS:
zeek/dns test-dns.log:
[0] parsing field value failed: the IP "40.126.31.143" is not one of the allowed test IPs (see: https://github.com/elastic/elastic-package/blob/main/internal/fields/_static/allowed_geo_ips.txt)
zeek/known_certs test-known_certs.log:
[0] parsing field value failed: field "event.kind"'s value "info" is not one of the allowed values (alert, enrichment, event, metric, state, pipeline_error, signal)
zeek/known_hosts test-known_hosts.log:
[0] parsing field value failed: field "event.kind"'s value "info" is not one of the allowed values (alert, enrichment, event, metric, state, pipeline_error, signal)
zeek/known_services test-known_services.log:
[0] parsing field value failed: field "event.kind"'s value "info" is not one of the allowed values (alert, enrichment, event, metric, state, pipeline_error, signal)
zeek/software test-software.log:
[0] parsing field value failed: field "event.kind"'s value "info" is not one of the allowed values (alert, enrichment, event, metric, state, pipeline_error, signal)

packages/zeek/data_stream/known_certs/elasticsearch/ingest_pipeline/default.yml

efd6 · 2022-06-15T03:02:25Z

The non-progress issue comes down to the absence of terminating new-lines in the test inputs. There are bunch of other inputs that suffer the same issue, but these all have more than one test line, so they pass because only a single event is required to allow a system test to pass (they would have been found if this were in place).

legoguy1000 · 2022-06-15T03:05:46Z

🤦‍♂️ ya I've had this issue before, good catch. I'll fix this in the AM and then I think we should be good.

efd6 · 2022-06-15T03:27:48Z

The other inputs that have the issue are

sample_logs/capture_loss.log
sample_logs/dhcp.log
sample_logs/files.log
sample_logs/notice.log
sample_logs/sip.log
sample_logs/snmp.log
sample_logs/ssh.log
sample_logs/ssl.log

legoguy1000 · 2022-06-15T12:59:13Z

@efd6 fixed

efd6 · 2022-06-15T22:14:33Z

/test

legoguy1000 · 2022-06-16T00:50:10Z

@efd6 I think i got it now 🤦

efd6 · 2022-06-16T01:47:57Z

That will fix the non-progress, but host is already populated. You will need to move the host assignments to other fields. I'm not sure where at this stage. I'm looking into this.

efd6 · 2022-06-16T01:48:01Z

/test

legoguy1000 · 2022-06-16T01:55:41Z

The forwarded tag needs to be added so that host is only set by zeek, not by the agent.

packages/zeek/data_stream/known_certs/_dev/test/system/test-logs-config.yml

packages/zeek/data_stream/known_hosts/_dev/test/system/test-logs-config.yml

packages/zeek/data_stream/known_services/_dev/test/system/test-logs-config.yml

packages/zeek/data_stream/software/_dev/test/system/test-logs-config.yml

efd6 · 2022-06-28T01:48:36Z

/test

legoguy1000 added 5 commits May 11, 2022 15:16

Add Software Log

e3c16d0

Add Known Certs

9e5e007

Add Known Hosts

58141c1

Add Known Services

7fdfff9

add samples

764a4f8

legoguy1000 requested a review from a team as a code owner May 11, 2022 15:46

update changelog

5aeba45

jamiehynds added the Team:Security-External Integrations label May 12, 2022

jamiehynds added Integration:zeek Zeek enhancement New feature or request labels May 12, 2022

efd6 reviewed May 16, 2022

View reviewed changes

Updates per comments

7a93917

fixed spelling

d9d019a

legoguy1000 requested a review from efd6 June 13, 2022 12:59

efd6 reviewed Jun 15, 2022

View reviewed changes

packages/zeek/data_stream/known_certs/elasticsearch/ingest_pipeline/default.yml Show resolved Hide resolved

fix errors

6864493

Merge remote-tracking branch 'upstream/main' into zeek-additional-logs

031b6dc

fix dataset

b6301d8

efd6 reviewed Jun 16, 2022

View reviewed changes

Default to forwarded tag to not add host fields to Zeek data

e3c1028

legoguy1000 requested a review from efd6 June 28, 2022 00:59

efd6 approved these changes Jun 28, 2022

View reviewed changes

efd6 merged commit a89fe94 into elastic:main Jun 28, 2022

legoguy1000 deleted the zeek-additional-logs branch June 28, 2022 12:00

andrewkroh mentioned this pull request Jun 30, 2022

[production] Promote many packages from snapshot elastic/package-storage#4814

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Zeek] Add additional data sets #3340

[Zeek] Add additional data sets #3340

legoguy1000 commented May 11, 2022 •

edited

Loading

elasticmachine commented May 11, 2022 •

edited

Loading

Build stats

`Google Storage Download`

`Error signal`

elasticmachine commented May 12, 2022 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented May 12, 2022

efd6 commented May 16, 2022

legoguy1000 commented May 20, 2022

efd6 commented May 20, 2022

elasticmachine commented May 20, 2022 •

edited

Loading

efd6 commented May 20, 2022

legoguy1000 commented May 27, 2022

efd6 commented Jun 14, 2022

legoguy1000 commented Jun 14, 2022

efd6 left a comment

efd6 commented Jun 15, 2022

legoguy1000 commented Jun 15, 2022

efd6 commented Jun 15, 2022

legoguy1000 commented Jun 15, 2022

efd6 commented Jun 15, 2022

legoguy1000 commented Jun 16, 2022

efd6 commented Jun 16, 2022 •

edited

Loading

efd6 commented Jun 16, 2022

legoguy1000 commented Jun 16, 2022

efd6 commented Jun 28, 2022

[Zeek] Add additional data sets #3340

[Zeek] Add additional data sets #3340

Conversation

legoguy1000 commented May 11, 2022 • edited Loading

What does this PR do?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

elasticmachine commented May 11, 2022 • edited Loading

❕ Build Aborted

Build stats

Steps errors

Google Storage Download

Error signal

🤖 GitHub comments

elasticmachine commented May 12, 2022 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented May 12, 2022

efd6 commented May 16, 2022

legoguy1000 commented May 20, 2022

efd6 commented May 20, 2022

elasticmachine commented May 20, 2022 • edited Loading

🌐 Coverage report

efd6 commented May 20, 2022

legoguy1000 commented May 27, 2022

efd6 commented Jun 14, 2022

legoguy1000 commented Jun 14, 2022

efd6 left a comment

Choose a reason for hiding this comment

efd6 commented Jun 15, 2022

legoguy1000 commented Jun 15, 2022

efd6 commented Jun 15, 2022

legoguy1000 commented Jun 15, 2022

efd6 commented Jun 15, 2022

legoguy1000 commented Jun 16, 2022

efd6 commented Jun 16, 2022 • edited Loading

efd6 commented Jun 16, 2022

legoguy1000 commented Jun 16, 2022

efd6 commented Jun 28, 2022

legoguy1000 commented May 11, 2022 •

edited

Loading

elasticmachine commented May 11, 2022 •

edited

Loading

`Google Storage Download`

`Error signal`

elasticmachine commented May 12, 2022 •

edited

Loading

elasticmachine commented May 20, 2022 •

edited

Loading

efd6 commented Jun 16, 2022 •

edited

Loading