v1.9.6 PROC'd Runners #431

sfc-gh-afedorov · 2020-08-28T00:51:43Z

Alerts

Alert Query and Suppression Runners, as well as Processor have largely been moved to SQL + JS Stored Procedures. This paves the way for deprecating Python and Docker for alert processing in a later version.

./run all now runs data connectors before violations and violations before alerts
deprecated baseline runners and scripts are removed (cf3bc6d, 972c5cd)
a rule may now declare a single handler without wrapping it in an array (972c5cd)

Data Connectors

fix bug in AWSIC running on latest EKS in Upgrade dependencies to enable SnowAlert to run on EKS with IRSA #425 (ty @edulop91)
AWSIC now records config describe-configuration-recorders results for all available regions (2844b94)
AWSIC now respects AWS rate limits for Get requests (2844b94)
AWSIC now handles ServerTimeout errors (no response in 60 seconds) gracefully (2cf48eb)
fix bug in Jira correlation logic
add custom Jira starting status via environment variable JIRA_STARTING_STATUS
JAMF and AzIC scheduling code are moved to generic system in table comments in 118b073, ...
fix Azure log to work for with additional log types in 947c394 (ty for Operation Logs from Azure Active Directory connector not seen in Snowflake table #414 @Chaitali-Sonparote)
minor cleanup of AzIC in 5ccc0f4
minor fix from gsuite API change e8a58e5
Okta connector can now use a custom domain and includes a pack for initial data cleanup
Jamf now handles large inventory sizes better in 5e55b8e

Packs

Bug fix in snowflake_security_monitoring in a3ad191 (ty to Intact Financial Corporation for the report & fix)
Basic Okta structures around ingested data
ZenGRC ingestion via external functions

Handlers

fix bug in Jira correlation logic [dc] Fix Jira handler correlation logic #424
rules can now send arbitrary payloads to ServiceNow handler in bbbb4c2
Jira handler works with single string source as well as list of multiple sources, and can now link alerts types to a triage repository (2d345aa)
SMTP handler can now pass host, user, port, and password as params (d452139)

WebUI

fix minor UX bugs and bump dependencies with security detections (425cdb6)

Deprecate Ingestion Scripts

ZenGRC is decommissioned and will be re-introduced as a pack in a future version (Zengrc pack #436)
Agari have been decommissioned without plans for re-introduction (79c3702)

also removes some vestigial code and logic

can't add default CURRENT_DATETIME to a column without recreating table

instead of guessing the error, log it explicitly

- all envars start with SA_ - params don't start with slack_

replaced with external functions

- standardize on SA_ prefix - add SA_JIRA_API_TOKEN alias for pwd

standardize on "SA_" prefix and overridable parameters

- more sensible run order for "all" param - fix running all connections sans ingest

this was deprecated by data connectors awhile ago

- fix Jira bug after status changes - link triage and alerts *after* escaping ticket content - re-enable Jira tests

sfc-gh-afedorov and others added 30 commits August 4, 2020 13:43

[dc] Remove vestigial column is AzIC

5ccc0f4

[dc] Remove manual Jamf schedule

118b073

Update azure_log.py

947c394

Bump version to v1.9.6

75eb368

[handler] adds custom payload to sn

bbbb4c2

[dc] improve logging in runner

be4b06c

[dc] Fix gsuite logs connector

e8a58e5

[dc] Fix Jira handler correlation logic (#424)

6638e9f

also removes some vestigial code and logic

Upgrade dependencies (#425)

27b0fcc

[dc, pack] Custom Okta domain and pack

77c6423

[dc] Keep Okta backwards compatible

d23bb3b

can't add default CURRENT_DATETIME to a column without recreating table

[tests] Fix minor bug in teardown

45a576b

[db] Log error in violations runner

8e7500f

instead of guessing the error, log it explicitly

[handlers] Standardize Slack envars / params

d6d50ee

- all envars start with SA_ - params don't start with slack_

[ingest] Remove Zengrc

4475ff3

replaced with external functions

[handler] Minor fixes to Jira handler

ed487cf

- standardize on SA_ prefix - add SA_JIRA_API_TOKEN alias for pwd

[handlers] Standardize SMTP handler

d452139

standardize on "SA_" prefix and overridable parameters

[run] Minor improvements to runners

c4fdeb5

- more sensible run order for "all" param - fix running all connections sans ingest

[handlers] Add single handler use-case

972c5cd

[run] Remove baselines runner

cf3bc6d

[dc] Fix Okta DC to include sortOrder

bd443e9

[dc] Jamf improvements

5e55b8e

[AQR] Default to all alerts

7d23674

Update run.py

03b022e

[dc] Add regions to config recorders

045314e

[dc] Tune AWSIC to AWS rate limit

2844b94

[dc] Fix AWSIC to record ServerTimeout errors

2cf48eb

[dc] Adds logic for custom collect_apis

8dd630b

[ingest] Removes agari and ingest_runner.py

79c3702

this was deprecated by data connectors awhile ago

[setup] Add non-install deps

3f6821e

[handler] Minor fixes in Jira handler

2d345aa

sfc-gh-afedorov changed the title ~~V1.9.6~~ v1.9.6 PROC'd Runners Sep 12, 2020

sfc-gh-afedorov and others added 7 commits September 14, 2020 09:40

Update snowflake_security_monitoring.sql

a3ad191

[dc] Remove Azure Collect schedule logic

09712a4

[run] Add aliases for connectors

f7734fc

fix python dev deps

5b0dcb8

Fixes broken Jira handlers

c45116b

[handlers] Fix Jira handler and add tests (#437)

94e91a5

- fix Jira bug after status changes - link triage and alerts *after* escaping ticket content - re-enable Jira tests

[WebUI] Fixes and dependency updates

425cdb6

sfc-gh-afedorov merged commit 368cbdc into master Sep 23, 2020

sfc-gh-afedorov deleted the v1.9.6 branch September 23, 2020 19:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

v1.9.6 PROC'd Runners #431

v1.9.6 PROC'd Runners #431

sfc-gh-afedorov commented Aug 28, 2020 •

edited

Loading

v1.9.6 PROC'd Runners #431

v1.9.6 PROC'd Runners #431

Conversation

sfc-gh-afedorov commented Aug 28, 2020 • edited Loading

Alerts

Data Connectors

Packs

Handlers

WebUI

Deprecate Ingestion Scripts

sfc-gh-afedorov commented Aug 28, 2020 •

edited

Loading