[Filebeat] Add ThreatQuotient to Threat Intel Module #27423 #27474

wanusmaximus · 2021-08-18T17:33:22Z

What does this PR do?

This PR adds the integration with the ThreatQ Threat Intel Platform to export security indicators from ThreatQ to Elastic.

Why is it important?

This is a new integration within the threatintel module.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

Closes [Filebeat] Add ThreatQuotient to Threat Intel Module #27423

cla-checker-service · 2021-08-18T17:33:26Z

💚 CLA has been signed

elasticmachine · 2021-08-18T17:38:18Z

❕ Build Aborted

The PR is not allowed to run in the CI yet

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Reason: The PR is not allowed to run in the CI yet
Start Time: 2021-08-18T17:33:36.987+0000
Duration: 4 min 27 sec
Commit: 7aa3d0c6d90f7f864888db96604cc5ee4d8a3b7f

Trends 🧪

Steps errors

Expand to view the steps failures

`Load a resource file from a shared library`

Took 0 min 0 sec . View more details on here
Description: approval-list/elastic/beats.yml

`Error signal`

Took 0 min 0 sec . View more details on here
Description: githubPrCheckApproved: The PR is not allowed to run in the CI yet. (Only users with write permissions can do so.)

Log output

Expand to view the last 100 lines of log output

[2021-08-18T17:35:47.779Z]  > git config --get remote.origin.url # timeout=10
[2021-08-18T17:35:47.784Z] using GIT_SSH to set credentials GitHub user @elasticmachine SSH key
[2021-08-18T17:35:47.787Z]  > git merge 9defc4acc11c3d6dd3539ce8289fbf938ade760f # timeout=10
[2021-08-18T17:35:47.800Z]  > git rev-parse HEAD^{commit} # timeout=10
[2021-08-18T17:35:47.810Z]  > git config core.sparsecheckout # timeout=10
[2021-08-18T17:35:47.814Z]  > git checkout -f 7aa3d0c6d90f7f864888db96604cc5ee4d8a3b7f # timeout=15
[2021-08-18T17:35:53.671Z] Commit message: "Updated changelog"
[2021-08-18T17:35:53.671Z] First time build. Skipping changelog.
[2021-08-18T17:35:53.671Z] Cleaning workspace
[2021-08-18T17:35:53.899Z]  > git --version # timeout=10
[2021-08-18T17:35:53.902Z]  > git --version # 'git version 2.17.1'
[2021-08-18T17:35:53.907Z] fatal: bad object d11dc1f0af4927b4f3a9d92a4bff0584b7734b6b
[2021-08-18T17:35:53.673Z]  > git rev-parse --verify HEAD # timeout=10
[2021-08-18T17:35:53.677Z] Resetting working tree
[2021-08-18T17:35:53.677Z]  > git reset --hard # timeout=10
[2021-08-18T17:35:53.764Z]  > git clean -fdx # timeout=10
[2021-08-18T17:35:54.934Z] Timeout set to expire in 4 hr 0 min
[2021-08-18T17:35:54.964Z] The timestamps step is unnecessary when timestamps are enabled for all Pipeline builds.
[2021-08-18T17:35:55.206Z] [INFO] Number of builds to be searched 10
[2021-08-18T17:35:55.880Z] [INFO] 'shallow' is forced to be disabled when running on PullRequests
[2021-08-18T17:35:55.905Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-27474@tmp
[2021-08-18T17:35:55.946Z] [INFO] gitCheckout: Checkout SCM PR-27474 with default customisation from the Item.
[2021-08-18T17:35:55.986Z] [INFO] Override default checkout
[2021-08-18T17:35:56.055Z] Sleeping for 10 sec
[2021-08-18T17:36:06.091Z] The recommended git tool is: git
[2021-08-18T17:36:06.200Z] using credential f6c7695a-671e-4f4f-a331-acdce44ff9ba
[2021-08-18T17:36:06.207Z] Wiping out workspace first.
[2021-08-18T17:36:06.220Z] Cloning the remote Git repository
[2021-08-18T17:36:06.220Z] Using shallow clone with depth 10
[2021-08-18T17:36:06.220Z] Avoid fetching tags
[2021-08-18T17:36:06.286Z] Cloning repository [email protected]:elastic/beats.git
[2021-08-18T17:36:06.368Z]  > git init /var/lib/jenkins/workspace/Beats_beats_PR-27474@tmp # timeout=10
[2021-08-18T17:36:06.373Z] Fetching upstream changes from [email protected]:elastic/beats.git
[2021-08-18T17:36:06.373Z]  > git --version # timeout=10
[2021-08-18T17:36:06.377Z]  > git --version # 'git version 2.17.1'
[2021-08-18T17:36:06.377Z] using GIT_SSH to set credentials GitHub user @elasticmachine SSH key
[2021-08-18T17:36:06.382Z]  > git fetch --no-tags --progress -- [email protected]:elastic/beats.git +refs/heads/*:refs/remotes/origin/* # timeout=15
[2021-08-18T17:36:29.624Z] Cleaning workspace
[2021-08-18T17:36:29.640Z] Using shallow fetch with depth 10
[2021-08-18T17:36:29.640Z] Pruning obsolete local branches
[2021-08-18T17:36:29.605Z]  > git config remote.origin.url [email protected]:elastic/beats.git # timeout=10
[2021-08-18T17:36:29.610Z]  > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
[2021-08-18T17:36:29.618Z]  > git config remote.origin.url [email protected]:elastic/beats.git # timeout=10
[2021-08-18T17:36:29.626Z]  > git rev-parse --verify HEAD # timeout=10
[2021-08-18T17:36:29.631Z] No valid HEAD. Skipping the resetting
[2021-08-18T17:36:29.632Z]  > git clean -fdx # timeout=10
[2021-08-18T17:36:29.643Z] Fetching upstream changes from [email protected]:elastic/beats.git
[2021-08-18T17:36:29.643Z] using GIT_SSH to set credentials GitHub user @elasticmachine SSH key
[2021-08-18T17:36:29.647Z]  > git fetch --no-tags --progress --prune -- [email protected]:elastic/beats.git +refs/pull/27474/head:refs/remotes/origin/PR-27474 +refs/heads/master:refs/remotes/origin/master # timeout=15
[2021-08-18T17:36:30.578Z] Merging remotes/origin/master commit 9defc4acc11c3d6dd3539ce8289fbf938ade760f into PR head commit 7aa3d0c6d90f7f864888db96604cc5ee4d8a3b7f
[2021-08-18T17:36:30.581Z]  > git config core.sparsecheckout # timeout=10
[2021-08-18T17:36:30.585Z]  > git checkout -f 7aa3d0c6d90f7f864888db96604cc5ee4d8a3b7f # timeout=15
[2021-08-18T17:36:32.270Z] Merge succeeded, producing 7aa3d0c6d90f7f864888db96604cc5ee4d8a3b7f
[2021-08-18T17:36:32.271Z] Checking out Revision 7aa3d0c6d90f7f864888db96604cc5ee4d8a3b7f (PR-27474)
[2021-08-18T17:36:32.646Z] Commit message: "Updated changelog"
[2021-08-18T17:36:32.646Z] Cleaning workspace
[2021-08-18T17:36:33.080Z]  > git --version # timeout=10
[2021-08-18T17:36:33.084Z]  > git --version # 'git version 2.17.1'
[2021-08-18T17:36:33.087Z] fatal: bad object d11dc1f0af4927b4f3a9d92a4bff0584b7734b6b
[2021-08-18T17:36:33.969Z] Masking supported pattern matches of $GIT_USERNAME or $GIT_PASSWORD
[2021-08-18T17:36:32.244Z]  > git remote # timeout=10
[2021-08-18T17:36:32.247Z]  > git config --get remote.origin.url # timeout=10
[2021-08-18T17:36:32.250Z] using GIT_SSH to set credentials GitHub user @elasticmachine SSH key
[2021-08-18T17:36:32.253Z]  > git merge 9defc4acc11c3d6dd3539ce8289fbf938ade760f # timeout=10
[2021-08-18T17:36:32.265Z]  > git rev-parse HEAD^{commit} # timeout=10
[2021-08-18T17:36:32.274Z]  > git config core.sparsecheckout # timeout=10
[2021-08-18T17:36:32.278Z]  > git checkout -f 7aa3d0c6d90f7f864888db96604cc5ee4d8a3b7f # timeout=15
[2021-08-18T17:36:32.649Z]  > git rev-parse --verify HEAD # timeout=10
[2021-08-18T17:36:32.652Z] Resetting working tree
[2021-08-18T17:36:32.652Z]  > git reset --hard # timeout=10
[2021-08-18T17:36:33.004Z]  > git clean -fdx # timeout=10
[2021-08-18T17:36:34.988Z] + git fetch https://****:****@github.com/elastic/beats.git +refs/pull/*/head:refs/remotes/origin/pr/*
[2021-08-18T17:37:56.663Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-27474@tmp/.git
[2021-08-18T17:37:56.886Z] Archiving artifacts
[2021-08-18T17:37:58.233Z] + git rev-parse HEAD
[2021-08-18T17:37:58.639Z] + git rev-parse HEAD
[2021-08-18T17:37:58.961Z] + git rev-parse origin/pr/27474
[2021-08-18T17:37:59.022Z] [INFO] githubEnv: Found Git Build Cause: pr
[2021-08-18T17:37:59.284Z] Masking supported pattern matches of $GITHUB_TOKEN
[2021-08-18T17:38:00.440Z] [WARN] githubApiCall: The REST API call https://api.github.com/repos/elastic/beats/pulls/27474/reviews return 0 elements
[2021-08-18T17:38:00.505Z] [INFO] githubPrCheckApproved: Title: Threatq - User: wanusmaximus - Author Association: FIRST_TIME_CONTRIBUTOR
[2021-08-18T17:38:00.861Z] ERROR: githubPrCheckApproved: The PR is not allowed to run in the CI yet
[2021-08-18T17:38:00.861Z] ERROR: githubPrCheckApproved: The PR is not allowed to run in the CI yet. (Only users with write permissions can do so.)
[2021-08-18T17:38:00.954Z] [INFO] Let's stop build #1. The PR is not allowed to run in the CI yet
[2021-08-18T17:38:00.987Z] Sleeping for 5 sec
[2021-08-18T17:38:02.575Z] Stage "Lint" skipped due to earlier failure(s)
[2021-08-18T17:38:02.653Z] Stage "Build&Test" skipped due to earlier failure(s)
[2021-08-18T17:38:02.733Z] Stage "Extended" skipped due to earlier failure(s)
[2021-08-18T17:38:02.814Z] Stage "Packaging" skipped due to earlier failure(s)
[2021-08-18T17:38:02.896Z] Stage "Packaging-Pipeline" skipped due to earlier failure(s)
[2021-08-18T17:38:03.009Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-27474/src/github.com/elastic/beats
[2021-08-18T17:38:03.367Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-27474
[2021-08-18T17:38:03.685Z] [INFO] getVaultSecret: Getting secrets
[2021-08-18T17:38:03.734Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-08-18T17:38:04.718Z] + chmod 755 generate-build-data.sh
[2021-08-18T17:38:04.718Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-27474/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-27474/runs/1 ABORTED 267456
[2021-08-18T17:38:04.718Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-27474/runs/1/steps/?limit=10000 -o steps-info.json
[2021-08-18T17:38:04.718Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-27474/runs/1/tests/?status=FAILED -o tests-errors.json
[2021-08-18T17:38:04.969Z] Retry 1/3 exited 22, retrying in 1 seconds...
[2021-08-18T17:38:05.880Z] Retry 2/3 exited 22, retrying in 2 seconds...

elasticmachine · 2021-08-18T19:48:06Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

mergify · 2021-08-30T06:26:35Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b threatq upstream/threatq
git merge upstream/master
git push upstream threatq

wanusmaximus · 2021-09-16T14:32:40Z

@peasead I think I resolved the conflicts. Please let me know if there are any additional actions that I need to take on my side. Thanks!!!

peasead · 2021-09-20T22:28:57Z

@P1llus is the CHANGELOG.next.asciidoc something that can be fixed by the ingest team or is something for the submitter?

mergify · 2021-09-22T12:57:00Z

This pull request does not have a backport label. Could you fix it @wanusmaximus? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

P1llus · 2021-10-05T12:49:28Z

I will see if I can get this merged sometime this week

P1llus · 2021-10-05T13:20:40Z

@wanusmaximus Would it be okay with you that I might add a few commits to this PR, just to resolve some merge conflicts and maybe quickly resolve some small issues?
Just to save some time :)

wanusmaximus · 2021-10-06T15:08:19Z

@P1llus Absolutely! Please feel free to make any changes that you think are necessary. Thanks!

P1llus · 2021-10-06T17:57:14Z

@wanusmaximus It seems like we are not allowed to add to your PR as a maintainer, would you be able to turn this on?
I think it is related to the PR settings you have as a user:
https://docs.github.com/en/github/collaborating-with-pull-requests/working-with-forks/allowing-changes-to-a-pull-request-branch-created-from-a-fork

mergify · 2021-10-06T19:25:11Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b threatq upstream/threatq
git merge upstream/master
git push upstream threatq

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Aug 18, 2021

wanusmaximus changed the title ~~Threatq~~ [Filebeat] Add ThreatQuotient to Threat Intel Module #27423 Aug 18, 2021

peasead requested review from adriansr and P1llus August 18, 2021 19:47

peasead added the Team:Security-External Integrations label Aug 18, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Aug 18, 2021

peasead added Filebeat Filebeat module labels Aug 18, 2021

wanusmaximus added 2 commits September 16, 2021 10:22

[Filebeat] Add ThreatQuotient to Threat Intel Module elastic#27423

da92fd0

Updated changelog

4d5a12a

wanusmaximus force-pushed the threatq branch from 7aa3d0c to 4d5a12a Compare September 16, 2021 14:26

mergify bot added the backport-skip Skip notification from the automated backport with mergify label Sep 22, 2021

wanusmaximus marked this pull request as draft October 6, 2021 19:24

wanusmaximus marked this pull request as ready for review October 6, 2021 19:24

wanusmaximus marked this pull request as draft October 6, 2021 19:25

wanusmaximus closed this Oct 7, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Filebeat] Add ThreatQuotient to Threat Intel Module #27423 #27474

[Filebeat] Add ThreatQuotient to Threat Intel Module #27423 #27474

wanusmaximus commented Aug 18, 2021

cla-checker-service bot commented Aug 18, 2021 •

edited

Loading

elasticmachine commented Aug 18, 2021

Build stats

Trends 🧪

`Load a resource file from a shared library`

`Error signal`

elasticmachine commented Aug 18, 2021

mergify bot commented Aug 30, 2021

wanusmaximus commented Sep 16, 2021

peasead commented Sep 20, 2021

mergify bot commented Sep 22, 2021

P1llus commented Oct 5, 2021

P1llus commented Oct 5, 2021

wanusmaximus commented Oct 6, 2021

P1llus commented Oct 6, 2021

mergify bot commented Oct 6, 2021

[Filebeat] Add ThreatQuotient to Threat Intel Module #27423 #27474

[Filebeat] Add ThreatQuotient to Threat Intel Module #27423 #27474

Conversation

wanusmaximus commented Aug 18, 2021

What does this PR do?

Why is it important?

Checklist

Related issues

cla-checker-service bot commented Aug 18, 2021 • edited Loading

elasticmachine commented Aug 18, 2021

❕ Build Aborted

Build stats

Trends 🧪

Steps errors

Load a resource file from a shared library

Error signal

Log output

elasticmachine commented Aug 18, 2021

mergify bot commented Aug 30, 2021

wanusmaximus commented Sep 16, 2021

peasead commented Sep 20, 2021

mergify bot commented Sep 22, 2021

P1llus commented Oct 5, 2021

P1llus commented Oct 5, 2021

wanusmaximus commented Oct 6, 2021

P1llus commented Oct 6, 2021

mergify bot commented Oct 6, 2021

cla-checker-service bot commented Aug 18, 2021 •

edited

Loading

`Load a resource file from a shared library`

`Error signal`