Jim/1244 org filters #3274

jimduff-usds · 2021-12-01T16:31:03Z

This PR implements ticket #1244

A very detailed list of changes is in #1244, but here's a quick summary:

Introducing new filter types routingFilter and processingModeFilter.
All four filter types (jurisdictionalFilter, qualityFilter, routingFilter, processingModeFilter) now work in exactly the same way, with the exception that qualityFilter still allows the old reverseTheQualityFilter flag, to support secondary receivers.
You can now specify filter information at the Organization level, which will result in a lot of simplification.
There are now defaults for all filters. (Previously only qualityFilter had a default.

How to Test

Run a smoke test

I have changed the State of Ignorance in organizations.yml to have this whole-organizations-level filter:

  filters:
    - topic: covid-19
      jurisdictionalFilter: [ "matches(ordering_facility_state, IG)" ]

So simply running the ./prime --run end2end test is a good test of the new feature.

Print out some quality filter logging

As a side-effect of this work, I had to mess around a lot with our quality filter logging. You can test that this way:

# Create some dummy data across all receivers
./prime data --input-fake 100 --input-schema primedatainput/pdi-covid-19 --output ./junk/simplerep.csv
# Route via curl (which triggers the filters) just to confirm its copacetic:
 curl -X POST -H "client:simple_report" -H "content-type:text/csv" --data-binary "@./junk/simplerep.csv" "http://localhost:7071/api/reports"
# Then you go into simplerep.csv manually screw up the data by blanking out values for patient_dob.
# Then run this again:
 curl -X POST -H "client:simple_report" -H "content-type:text/csv" --data-binary "@./junk/simplerep.csv" "http://localhost:7071/api/reports"
# And you should see all kinds of wonderfully detailed quality filter stuff related to patient_dob.

Checklist

Other testing I did besides the list below

ran testIntegration local
ran the above suggested test
added a bunch of unit tests

Testing

Tested locally?
Ran ./prime test or ./gradlew testSmoke against local Docker ReportStream container?
(For Changes to /frontend-react/...) Ran npm run lint:write?
Added tests?

Process

[N] Are there licensing issues with any new dependencies introduced?
[Y] Includes a summary of what a code reviewer should test/verify?
[N] Updated the release notes? Need to do this, but the release notes are stuck in a different PR
Database changes are submitted as a separate PR?
DevOps team has been notified if PR requires ops support?

Fixes

Create Organization-level filters; make all filters work the same way #1244

To Be Done

See list in Create Organization-level filters; make all filters work the same way #1244

Pull reviewers stats

Stats for the last 30 days:

User	Total reviews	Median time to review	Total comments
MauriceReeves-usds	31	19h 27m	6
cwinters-usds	26	5h 21m	18
kevinhaube	19	32m	9
carlosfelix2	19	5h 3m	53
clediggins-usds	18	2h 10m	28
sean-usds	16	1h 30m	4
jimduff-usds	15	14h 29m	17
RickHawesUSDS	14	4h 15m	18
whytheplatypus	8	2h 23m	12
TomNUSDS	7	22m	2
jeremy-page	4	14m	0
brick-green-agile6	4	7h 3m	1
ahay-agile6	2	1d 9h 28m	0
jorg3lopez	2	17h 16m	0
jbiskie	2	1h 5m	0
JosiahSiegel	1	5d 2h 55m	2
MikeC-A6	1	7h 7m	3
rhood23699	1	14h 10m	0
sethdarragile6	1	2h 17m	0

Pull reviewers stats

Stats for the last 30 days:

User	Total reviews	Median time to review	Total comments
MauriceReeves-usds	28	17h 35m	6
cwinters-usds	25	5h 9m	18
kevinhaube	19	32m	9
carlosfelix2	18	10h 17m	51
clediggins-usds	17	1h 53m	28
sean-usds	16	1h 30m	4
RickHawesUSDS	14	4h 15m	18
jimduff-usds	14	14h 29m	17
whytheplatypus	8	2h 23m	12
TomNUSDS	7	22m	2
brick-green-agile6	4	7h 3m	1
ahay-agile6	2	1d 9h 28m	0
jorg3lopez	2	17h 16m	0
jeremy-page	2	8m	0
JosiahSiegel	1	5d 2h 55m	2
MikeC-A6	1	7h 7m	3
jbiskie	1	1h 44m	0
rhood23699	1	14h 10m	0
sethdarragile6	1	2h 17m	0

Pull reviewers stats

Stats for the last 30 days:

User	Total reviews	Median time to review	Total comments
MauriceReeves-usds	26	15h 21m	6
cwinters-usds	25	14h 13m	26
kevinhaube	18	28m	9
carlosfelix2	18	10h 17m	51
clediggins-usds	17	2h 10m	35
jimduff-usds	14	14h 9m	33
sean-usds	14	1h 20m	4
RickHawesUSDS	12	4h 17m	30
whytheplatypus	8	2h 23m	12
TomNUSDS	6	40m	3
brick-green-agile6	4	7h 3m	1
ahay-agile6	2	1d 9h 28m	0
jorg3lopez	2	17h 16m	0
JosiahSiegel	1	5d 2h 55m	2
MikeC-A6	1	7h 7m	3
jbiskie	1	1h 44m	0
rhood23699	1	14h 10m	0
sethdarragile6	1	2h 17m	0

Pull reviewers stats

Stats for the last 30 days:

User	Total reviews	Median time to review	Total comments
cwinters-usds	30	1h 50m	18
MauriceReeves-usds	26	14h 59m	6
kevinhaube	23	48m	10
sean-usds	21	1h 30m	6
carlosfelix2	19	16h 53m	54
jimduff-usds	18	1d 1h 24m	8
RickHawesUSDS	14	4h 51m	5
whytheplatypus	10	17h 22m	19
clediggins-usds	10	2h 51m	24
TomNUSDS	9	22m	2
jbiskie	5	1h 26m	1
ahay-agile6	2	1d 9h 28m	0
jorg3lopez	2	9d 21h 37m	0
brick-green-agile6	2	14m	0
JosiahSiegel	1	5d 2h 55m	2
MikeC-A6	1	7h 7m	3
rhood23699	1	14h 10m	0
sethdarragile6	1	2h 17m	0

…d routing filters

… into filterAndTranslateByReceiver and out of translateByReceiver

… Renamed some classes for clarity

github-actions · 2021-12-01T16:36:54Z

Unit Test Results

  41 files ±0   41 suites ±0 28s ⏱️ -1s
314 tests +5 310 ✔️ +5 4 💤 ±0 0 ❌ ±0

Results for commit 1fd4b0d. ± Comparison against base commit 583e5ea.

This pull request removes 14 and adds 19 tests. Note that renamed tests count towards both.

gov.cdc.prime.router.JurisdictionalFilterTests ‑ test DoesNotMatch()
gov.cdc.prime.router.JurisdictionalFilterTests ‑ test FilterByCounty()
gov.cdc.prime.router.JurisdictionalFilterTests ‑ test HasAtLeastOneOf()
gov.cdc.prime.router.JurisdictionalFilterTests ‑ test HasValidDataFor()
gov.cdc.prime.router.JurisdictionalFilterTests ‑ test IsValidCLIA()
gov.cdc.prime.router.JurisdictionalFilterTests ‑ test Matches with multiple regexi()
gov.cdc.prime.router.JurisdictionalFilterTests ‑ test Matches()
gov.cdc.prime.router.JurisdictionalFilterTests ‑ test OrEquals()
gov.cdc.prime.router.JurisdictionalFilterTests ‑ test allowAll()
gov.cdc.prime.router.JurisdictionalFilterTests ‑ test empty DoesNotMatch()
…

gov.cdc.prime.router.ReportStreamFilterDefinitionTests ‑ test DoesNotMatch()
gov.cdc.prime.router.ReportStreamFilterDefinitionTests ‑ test FilterByCounty()
gov.cdc.prime.router.ReportStreamFilterDefinitionTests ‑ test HasAtLeastOneOf()
gov.cdc.prime.router.ReportStreamFilterDefinitionTests ‑ test HasValidDataFor()
gov.cdc.prime.router.ReportStreamFilterDefinitionTests ‑ test IsValidCLIA()
gov.cdc.prime.router.ReportStreamFilterDefinitionTests ‑ test Matches with multiple regexi()
gov.cdc.prime.router.ReportStreamFilterDefinitionTests ‑ test Matches()
gov.cdc.prime.router.ReportStreamFilterDefinitionTests ‑ test OrEquals()
gov.cdc.prime.router.ReportStreamFilterDefinitionTests ‑ test allowAll()
gov.cdc.prime.router.ReportStreamFilterDefinitionTests ‑ test allowNone()
…

♻️ This comment has been updated with latest results.

github-actions · 2021-12-01T16:38:07Z

Integration Test Results

11 files ±0 11 suites ±0 1m 35s ⏱️ -3s
83 tests ±0 83 ✔️ ±0 0 💤 ±0 0 ❌ ±0

Results for commit 1fd4b0d. ± Comparison against base commit 583e5ea.

♻️ This comment has been updated with latest results.

…instead

…feature, for 'ignore'

jimduff-usds · 2021-12-05T17:51:06Z

prime-router/settings/organizations.yml

+  filters:
+    - topic: covid-19
+      jurisdictionalFilter: [ "matches(ordering_facility_state, IG)" ]


This is an example of the new organization-level filters. I only made this change for 'ignore', for now - we'll want to do this for all the receiver jurisdictions, as follow-on work.

jimduff-usds · 2021-12-05T20:11:38Z

prime-router/src/main/kotlin/Metadata.kt

-    private var jurisdictionalFilters = listOf(
+    private var reportStreamFilterDefinitions = listOf(


Throughout, the concept of filters has been generalized. The new general name is "ReportStreamFilters"

jimduff-usds · 2021-12-05T20:13:59Z

prime-router/src/main/kotlin/Report.kt

+     * @return an emptyList if the columnName is null (not an error)
+     *
+     */
+    fun getValuesInRows(columnName: String?, rows: List<Int>, default: String? = null): List<String> {


This helper function is general, but created for the specific purpose of pulling out the trackingElement IDs for the subset of rows that have been filtered out, so we can delivery good logging and error messages.

jimduff-usds · 2021-12-05T20:15:46Z

prime-router/src/main/kotlin/ReportStreamFilter.kt

+data class ReportStreamFilters(
+    val topic: String,
+    val jurisdictionalFilter: ReportStreamFilter?,
+    val qualityFilter: ReportStreamFilter?,
+    val routingFilter: ReportStreamFilter?,
+) {


I decided to make ReportStreamFilters a data class with specific fields, rather than, say, a Map from "filterType" -> "set of Filters", because it makes the yaml representation a little cleaner.

jimduff-usds · 2021-12-05T20:17:00Z

prime-router/src/main/kotlin/ReportStreamFilter.kt

+        val defaultCovid19QualityFilter: ReportStreamFilter = listOf(
+            // valid basic test info
+            "hasValidDataFor(message_id)",
+            "hasValidDataFor(equipment_model_name)",
+            "hasValidDataFor(specimen_type)",
+            "hasValidDataFor(test_result)",
+            // valid basic human info
+            "hasValidDataFor(patient_last_name, patient_first_name)",
+            "hasValidDataFor(patient_dob)",


These have been somewhat split apart into one function per field. This is a cheap way to get more fine-grained logging messages.

jimduff-usds · 2021-12-05T20:18:47Z

prime-router/src/main/kotlin/ReportStreamFilter.kt

+            jurisdictionalFilter = null,
+            qualityFilter = listOf("matches(a, no)"),
+            routingFilter = listOf("matches(b, false)")
+        )


You'll see three sets of filters here: One "real" one for covid-19, then a couple test filter sets.

Note that each ReportStreamFilters obj has to be associated with a 'topic', because filters surely vary by topic.

All of this belongs in Settings, not here - a known ticketed issue.

jimduff-usds · 2021-12-05T20:20:15Z

prime-router/src/main/kotlin/ReportStreamFilterDefinition.kt

+/**
+ * A filter that filter everything -- allows no data through.  Useful as a default to be overridden.
+ */
+class AllowNone : ReportStreamFilterDefinition {


The new AllowNone filter is required for the global default jurisdictionFilter.

jimduff-usds · 2021-12-05T20:24:11Z

prime-router/src/main/kotlin/ReportStreamFilterDefinition.kt

                    Selection.withRange(0, table.rowCount()), selection,
                    "$name(${args.joinToString(",")})", receiver, doAuditing
                )
            }
        }
        return selection
    }
-}
-


The code was moved

jimduff-usds · 2021-12-05T20:24:47Z

prime-router/src/main/kotlin/Translator.kt

-                val mappedReport = translateByReceiver(jurisFilteredReport, receiver, defaultValues)
-                Pair(mappedReport, receiver)
+                // Filter the report
+                val filteredReport = filterByAllFilterTypes(settings, input, receiver) ?: return@mapNotNull null


This is the key call to the new code here.

jimduff-usds · 2021-12-05T20:25:57Z

prime-router/src/main/kotlin/Translator.kt

-            val (fnName, fnArgs) = JurisdictionalFilters.parseJurisdictionalFilter(filterSpec)
-            val filterFn = metadata.findJurisdictionalFilter(fnName)
-                ?: error("JurisdictionalFilter $fnName is not found")
+    fun filterByAllFilterTypes(settings: SettingsProvider, input: Report, receiver: Receiver): Report? {


The main new code is these two new methods.

…s own filter

jimduff-usds · 2021-12-06T04:34:54Z

prime-router/src/main/kotlin/ReportStreamFilter.kt

+            topic = "covid-19",
+            jurisdictionalFilter = listOf("allowNone()"), // Receiver *must* override this to get data!
+            qualityFilter = defaultCovid19QualityFilter,
+            routingFilter = listOf("allowAll()"),
+            processingModeFilter = listOf("doesNotMatch(processing_mode_code, T, D)"), // No Training/Debug data


Note the new default jurisdictionalFilter is to allow zilch. This is a good safety/quality improvement.

Note that the routingFilter defaults to allowing everything - no worried about inadvertently overriding an important default.

And the processingModeFilter is set globally to NOT routing Training and Debug data. Nice to not have to specify this over and over. And it didn't really belong in the quality filter. This now allows you to safely have a secondary feed that is 'reverseTheQualityFilter=true' without mistakenly getting 'T' and 'D' data as well. Another safety/quality improvement.

brick-green · 2021-12-06T22:15:10Z

prime-router/src/main/kotlin/Receiver.kt

- * @param qualityFilter defines the set of elements and regexes that do qualiyty filtering on the data for this receiver
+ * @param jurisdictionalFilter defines the geographic region filters for this receiver
+ * @param qualityFilter defines the filters that remove data, based on quality criteria
+ * @param routingFilter defines the filters that remove data the receiver does not want, based on who sent it.


Since the routingFilter will be used for more than just sender id perhaps the description should be edited to make it clear that any field in the report can be used.

prime-router/src/main/kotlin/ReportStreamFilterDefinition.kt

cwinters-usds · 2021-12-09T21:31:37Z

.environment/gitleaks/gitleaks-config.toml

@@ -120,6 +120,7 @@ title = "PRIME ReportStream Gitleaks Configuration"
    ]
    [rules.allowlist]
        regexes = [
+            'api/lookuptables/list',


cwinters-usds

I don't know enough about the current behavior to judge the filtering changes, but the one change devops needs to change (to the gitleaks config) looks ok.

prime-router/src/main/kotlin/Receiver.kt

sonarqubecloud · 2021-12-13T13:59:55Z

SonarCloud Quality Gate failed.

0 Bugs
0 Vulnerabilities
0 Security Hotspots
27 Code Smells

No Coverage information
18.3% Duplication

* 3125 Moved fips-county table to the database (#3126) * Moved fips-county table to the database * Fixed unit tests * Run integration tests with container * Clear files after upload (#3355) * Clear files after upload * Change FileInput key to force reset * Cleanup * Correct reversed upload errors and warnings (#3359) * added null check on originState * add unit test for cliaForOutOFStateTesting * Bump cypress from 9.1.0 to 9.1.1 in /frontend-react (#3317) Bumps [cypress](https://github.com/cypress-io/cypress) from 9.1.0 to 9.1.1. - [Release notes](https://github.com/cypress-io/cypress/releases) - [Changelog](https://github.com/cypress-io/cypress/blob/develop/.releaserc.base.js) - [Commits](cypress-io/cypress@v9.1.0...v9.1.1) --- updated-dependencies: - dependency-name: cypress dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * VT - replaceValue for MSH-3-1 (#3370) * MSH-3-1 added to HD field list * replace MSH-3-1 value * adds test case * edits to serializer file * ignored test case until it can be ran * comments for ignored test case * 3271 Added wait for API to be available to settings command (#3337) * Added wait for API to settings * Update Zip Codes (#3376) * Fix 404 page copy for staging and production (#3377) Since we renamed "build" -> "build:staging" and "build:production", the `"postbuild"` should also be renamed to "postbuild:staging" and "postbuild:production" * addd tests for missing state, only provider state, and only ordering state * 343 - Fixed the ValueSets Code: ">" for Markdown Table (#3204) * 343 - Fixed the ValueSets Code: ">" for Markdown Table Code: ">" in common.valuesets table hl70078 is the blockquote markdown control character. We need to user ">" (html code symbol for ">" instead. * Added Unit Test case for checking the > (special character for >) * How It Works bad path fix (#3384) * Fixes 404 for How It Works and handles non-extended HIW paths * adjusted the strings to conform with ktlint length * changed away from sting builder * Create FORM-NAME.yml * fix errror with multiline string * Bump org.flywaydb.flyway from 8.0.4 to 8.2.0 in /prime-router (#3261) Bumps org.flywaydb.flyway from 8.0.4 to 8.2.0. --- updated-dependencies: - dependency-name: org.flywaydb.flyway dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Maurice Reeves <[email protected]> * Bump ktorVersion from 1.6.4 to 1.6.7 in /prime-router (#3353) Bumps `ktorVersion` from 1.6.4 to 1.6.7. Updates `ktor-client-core` from 1.6.4 to 1.6.7 - [Release notes](https://github.com/ktorio/ktor/releases) - [Changelog](https://github.com/ktorio/ktor/blob/main/CHANGELOG.md) - [Commits](https://github.com/ktorio/ktor/commits) Updates `ktor-client-cio` from 1.6.4 to 1.6.7 - [Release notes](https://github.com/ktorio/ktor/releases) - [Changelog](https://github.com/ktorio/ktor/blob/main/CHANGELOG.md) - [Commits](https://github.com/ktorio/ktor/commits) Updates `ktor-client-apache` from 1.6.4 to 1.6.7 - [Release notes](https://github.com/ktorio/ktor/releases) - [Changelog](https://github.com/ktorio/ktor/blob/main/CHANGELOG.md) - [Commits](https://github.com/ktorio/ktor/commits) Updates `ktor-client-logging` from 1.6.4 to 1.6.7 - [Release notes](https://github.com/ktorio/ktor/releases) - [Changelog](https://github.com/ktorio/ktor/blob/main/CHANGELOG.md) - [Commits](https://github.com/ktorio/ktor/commits) Updates `ktor-client-mock` from 1.6.4 to 1.6.7 - [Release notes](https://github.com/ktorio/ktor/releases) - [Changelog](https://github.com/ktorio/ktor/blob/main/CHANGELOG.md) - [Commits](https://github.com/ktorio/ktor/commits) --- updated-dependencies: - dependency-name: io.ktor:ktor-client-core dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: io.ktor:ktor-client-cio dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: io.ktor:ktor-client-apache dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: io.ktor:ktor-client-logging dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: io.ktor:ktor-client-mock dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Maurice Reeves <[email protected]> * Bump azure-core from 1.22.0 to 1.23.1 in /prime-router (#3352) Bumps [azure-core](https://github.com/Azure/azure-sdk-for-java) from 1.22.0 to 1.23.1. - [Release notes](https://github.com/Azure/azure-sdk-for-java/releases) - [Commits](Azure/azure-sdk-for-java@azure-core_1.22.0...azure-core_1.23.1) --- updated-dependencies: - dependency-name: com.azure:azure-core dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Maurice Reeves <[email protected]> * Bump junit-jupiter-api from 5.8.1 to 5.8.2 in /prime-router (#3263) Bumps [junit-jupiter-api](https://github.com/junit-team/junit5) from 5.8.1 to 5.8.2. - [Release notes](https://github.com/junit-team/junit5/releases) - [Commits](junit-team/junit5@r5.8.1...r5.8.2) --- updated-dependencies: - dependency-name: org.junit.jupiter:junit-jupiter-api dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Rick Hawes <[email protected]> Co-authored-by: Maurice Reeves <[email protected]> * Bump thymeleaf from 3.0.12.RELEASE to 3.0.14.RELEASE in /prime-router (#3354) Bumps [thymeleaf](https://github.com/thymeleaf/thymeleaf) from 3.0.12.RELEASE to 3.0.14.RELEASE. - [Release notes](https://github.com/thymeleaf/thymeleaf/releases) - [Changelog](https://github.com/thymeleaf/thymeleaf/blob/3.0-master/ChangeLog.txt) - [Commits](thymeleaf/thymeleaf@thymeleaf-3.0.12.RELEASE...thymeleaf-3.0.14.RELEASE) --- updated-dependencies: - dependency-name: org.thymeleaf:thymeleaf dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Maurice Reeves <[email protected]> * consolidated variables * removed trimIndent * Fix unit tests (#3389) - Fix all broken unit tests * Added check for last modified time to settings CLI (#3403) * Added check for last modified time to settings CLI * Run automated checks on regular PR (NOT master or production) (#3404) * Run automated checks on regular PR (NOT master or production) * Remove unused includes * Updated gitleaks version, commands, and flags (#3397) * ...commit actually created by Matt Armstrong Co-authored-by: Chris Winters <[email protected]> * Remove the comment I accidentally kept in with previous (#3409) * Update log4j due to security vulnerability (#3408) * More about vulnerability: https://www.lunasec.io/docs/blog/log4j-zero-day/ * #3347 Allow specifying org for submission history (#3356) * Allow specifying org for submission history if authorized to see that organization * Improve queryParam parsing * Update submission smoke test url path Co-authored-by: sean pennino <[email protected]> * Jim/2314 payload name (#3201) * Implemented new payloadName URL parameter * Jim/1244 org filters (#3274) * Rearchitected ReportStream's Filters: Created Organization-level filters, new routingFilter, new processingMode filter, and defaults for all four filter types. * Update issue templates Added task to update customerStatus in settings * Removed entropy scan (#3435) Co-authored-by: Chris Winters <[email protected]> * Fixing 2 bugs with token request (#3439) * Admins can view submission history (#3441) * Initial Submissions page setup * Basic submissions fetch * Submissions table * Lint fixes * Add title * useState for submissions * Bump @okta/okta-auth-js from 5.9.0 to 5.9.1 in /frontend-react (#3418) Bumps [@okta/okta-auth-js](https://github.com/okta/okta-auth-js) from 5.9.0 to 5.9.1. - [Release notes](https://github.com/okta/okta-auth-js/releases) - [Changelog](https://github.com/okta/okta-auth-js/blob/master/CHANGELOG.md) - [Commits](okta/okta-auth-js@okta-auth-js-5.9.0...okta-auth-js-5.9.1) --- updated-dependencies: - dependency-name: "@okta/okta-auth-js" dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: sean pennino <[email protected]> * Bump typescript from 4.5.2 to 4.5.3 in /frontend-react (#3422) Bumps [typescript](https://github.com/Microsoft/TypeScript) from 4.5.2 to 4.5.3. - [Release notes](https://github.com/Microsoft/TypeScript/releases) - [Commits](microsoft/TypeScript@v4.5.2...v4.5.3) --- updated-dependencies: - dependency-name: typescript dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: sean pennino <[email protected]> * Add useEffect to prevent unnecessary submission fetches (#3459) * Remove unncessary JSON parse * Add useEffect * Bump azure-core-http-netty from 1.11.2 to 1.11.4 in /prime-router (#3396) Bumps [azure-core-http-netty](https://github.com/Azure/azure-sdk-for-java) from 1.11.2 to 1.11.4. - [Release notes](https://github.com/Azure/azure-sdk-for-java/releases) - [Commits](Azure/azure-sdk-for-java@azure-core-http-netty_1.11.2...azure-core-http-netty_1.11.4) --- updated-dependencies: - dependency-name: com.azure:azure-core-http-netty dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Maurice Reeves <[email protected]> * Bump bcpkix-jdk15on from 1.69 to 1.70 in /prime-router (#3398) Bumps [bcpkix-jdk15on](https://github.com/bcgit/bc-java) from 1.69 to 1.70. - [Release notes](https://github.com/bcgit/bc-java/releases) - [Changelog](https://github.com/bcgit/bc-java/blob/master/docs/releasenotes.html) - [Commits](https://github.com/bcgit/bc-java/commits) --- updated-dependencies: - dependency-name: org.bouncycastle:bcpkix-jdk15on dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Maurice Reeves <[email protected]> * Bump fuzzy-csv from 1.7.2 to 1.7.3 in /prime-router (#3400) Bumps [fuzzy-csv](https://github.com/kayr/fuzzy-csv) from 1.7.2 to 1.7.3. - [Release notes](https://github.com/kayr/fuzzy-csv/releases) - [Commits](kayr/fuzzy-csv@1.7.2...1.7.3) --- updated-dependencies: - dependency-name: com.github.kayr:fuzzy-csv dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Maurice Reeves <[email protected]> * Bump @types/testing-library__jest-dom in /frontend-react (#3420) Bumps [@types/testing-library__jest-dom](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/testing-library__jest-dom) from 5.14.1 to 5.14.2. - [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases) - [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/testing-library__jest-dom) --- updated-dependencies: - dependency-name: "@types/testing-library__jest-dom" dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: sean pennino <[email protected]> * Bump libphonenumber from 8.12.38 to 8.12.39 in /prime-router (#3401) Bumps [libphonenumber](https://github.com/google/libphonenumber) from 8.12.38 to 8.12.39. - [Release notes](https://github.com/google/libphonenumber/releases) - [Changelog](https://github.com/google/libphonenumber/blob/master/making-metadata-changes.md) - [Commits](google/libphonenumber@v8.12.38...v8.12.39) --- updated-dependencies: - dependency-name: com.googlecode.libphonenumber:libphonenumber dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Maurice Reeves <[email protected]> * Bump commonmark from 0.18.0 to 0.18.1 in /prime-router (#3402) Bumps [commonmark](https://github.com/commonmark/commonmark-java) from 0.18.0 to 0.18.1. - [Release notes](https://github.com/commonmark/commonmark-java/releases) - [Changelog](https://github.com/commonmark/commonmark-java/blob/main/CHANGELOG.md) - [Commits](commonmark/commonmark-java@commonmark-parent-0.18.0...commonmark-parent-0.18.1) --- updated-dependencies: - dependency-name: org.commonmark:commonmark dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Maurice Reeves <[email protected]> * Bump jvm from 1.6.0 to 1.6.10 in /prime-router (#3465) Bumps jvm from 1.6.0 to 1.6.10. --- updated-dependencies: - dependency-name: jvm dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Maurice Reeves <[email protected]> * Bump bcmail-jdk15on from 1.69 to 1.70 in /prime-router (#3466) Bumps [bcmail-jdk15on](https://github.com/bcgit/bc-java) from 1.69 to 1.70. - [Release notes](https://github.com/bcgit/bc-java/releases) - [Changelog](https://github.com/bcgit/bc-java/blob/master/docs/releasenotes.html) - [Commits](https://github.com/bcgit/bc-java/commits) --- updated-dependencies: - dependency-name: org.bouncycastle:bcmail-jdk15on dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Maurice Reeves <[email protected]> * Bump com.microsoft.azure.azurefunctions in /prime-router (#3467) Bumps com.microsoft.azure.azurefunctions from 1.8.1 to 1.8.2. --- updated-dependencies: - dependency-name: com.microsoft.azure.azurefunctions dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Maurice Reeves <[email protected]> * Bump org.flywaydb.flyway from 8.2.0 to 8.2.2 in /prime-router (#3470) Bumps org.flywaydb.flyway from 8.2.0 to 8.2.2. --- updated-dependencies: - dependency-name: org.flywaydb.flyway dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Maurice Reeves <[email protected]> * Bump kotlinVersion from 1.6.0 to 1.6.10 in /prime-router (#3473) Bumps `kotlinVersion` from 1.6.0 to 1.6.10. Updates `kotlin-stdlib-jdk8` from 1.6.0 to 1.6.10 - [Release notes](https://github.com/JetBrains/kotlin/releases) - [Changelog](https://github.com/JetBrains/kotlin/blob/v1.6.10/ChangeLog.md) - [Commits](JetBrains/kotlin@v1.6.0...v1.6.10) Updates `kotlin-stdlib-common` from 1.6.0 to 1.6.10 - [Release notes](https://github.com/JetBrains/kotlin/releases) - [Changelog](https://github.com/JetBrains/kotlin/blob/v1.6.10/ChangeLog.md) - [Commits](JetBrains/kotlin@v1.6.0...v1.6.10) Updates `kotlin-reflect` from 1.6.0 to 1.6.10 - [Release notes](https://github.com/JetBrains/kotlin/releases) - [Changelog](https://github.com/JetBrains/kotlin/blob/v1.6.10/ChangeLog.md) - [Commits](JetBrains/kotlin@v1.6.0...v1.6.10) --- updated-dependencies: - dependency-name: org.jetbrains.kotlin:kotlin-stdlib-jdk8 dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: org.jetbrains.kotlin:kotlin-stdlib-common dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: org.jetbrains.kotlin:kotlin-reflect dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Maurice Reeves <[email protected]> * Bump json from 20210307 to 20211205 in /prime-router (#3476) Bumps [json](https://github.com/douglascrockford/JSON-java) from 20210307 to 20211205. - [Release notes](https://github.com/douglascrockford/JSON-java/releases) - [Changelog](https://github.com/stleary/JSON-java/blob/master/docs/RELEASES.md) - [Commits](https://github.com/douglascrockford/JSON-java/commits) --- updated-dependencies: - dependency-name: org.json:json dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Maurice Reeves <[email protected]> * Bump bcprov-jdk15on from 1.69 to 1.70 in /prime-router (#3475) Bumps [bcprov-jdk15on](https://github.com/bcgit/bc-java) from 1.69 to 1.70. - [Release notes](https://github.com/bcgit/bc-java/releases) - [Changelog](https://github.com/bcgit/bc-java/blob/master/docs/releasenotes.html) - [Commits](https://github.com/bcgit/bc-java/commits) --- updated-dependencies: - dependency-name: org.bouncycastle:bcprov-jdk15on dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump sendgrid-java from 4.8.0 to 4.8.1 in /prime-router (#3474) Bumps [sendgrid-java](https://github.com/sendgrid/sendgrid-java) from 4.8.0 to 4.8.1. - [Release notes](https://github.com/sendgrid/sendgrid-java/releases) - [Changelog](https://github.com/sendgrid/sendgrid-java/blob/main/CHANGELOG.md) - [Commits](sendgrid/sendgrid-java@4.8.0...4.8.1) --- updated-dependencies: - dependency-name: com.sendgrid:sendgrid-java dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Maurice Reeves <[email protected]> * Bump as2-lib from 4.8.0 to 4.9.0 in /prime-router (#3478) Bumps [as2-lib](https://github.com/phax/as2-lib) from 4.8.0 to 4.9.0. - [Release notes](https://github.com/phax/as2-lib/releases) - [Commits](phax/as2-lib@as2-lib-parent-pom-4.8.0...as2-lib-parent-pom-4.9.0) --- updated-dependencies: - dependency-name: com.helger.as2:as2-lib dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Maurice Reeves <[email protected]> * Bump flyway-core from 8.2.0 to 8.2.2 in /prime-router (#3479) Bumps [flyway-core](https://github.com/flyway/flyway) from 8.2.0 to 8.2.2. - [Release notes](https://github.com/flyway/flyway/releases) - [Commits](flyway/flyway@flyway-8.2.0...flyway-8.2.2) --- updated-dependencies: - dependency-name: org.flywaydb:flyway-core dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Maurice Reeves <[email protected]> * Fixed bug that allowNone was missing from metadata list. Added unit test (#3444) * Fixes issue with the new truncation logic truncating fields it should not be (#3429) * Fixes issue with the new truncation logic truncating fields it should not be * Fixes typo * Removed gitleaks for now (#3483) * Update log4j dependency (again) due to refined handling of security vulnerability (#3484) Co-authored-by: sean pennino <[email protected]> Co-authored-by: Brick Green <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jorge Lopez <[email protected]> Co-authored-by: Rick Hood <[email protected]> Co-authored-by: TomNUSDS <[email protected]> Co-authored-by: OTT SATHNGAM <[email protected]> Co-authored-by: Kevin Haube <[email protected]> Co-authored-by: brick-green-agile6 <[email protected]> Co-authored-by: Anshul Kumar <[email protected]> Co-authored-by: Maurice Reeves <[email protected]> Co-authored-by: Rick Hawes <[email protected]> Co-authored-by: Matt Armstrong <[email protected]> Co-authored-by: Chris Winters <[email protected]> Co-authored-by: Chris Winters <[email protected]> Co-authored-by: David Gage <[email protected]> Co-authored-by: jimduff-usds <[email protected]> Co-authored-by: Joel Biskie <[email protected]>

jimduff-usds added 5 commits November 29, 2021 07:28

Created new FilterType concept

bed3440

juris filter defaults to 'off'

a1abc6e

implemented generic filter function that works for juris, quality, an…

d44cfa1

…d routing filters

Juris filter now defaults to allowNone(). Moved all various filtering…

df312c6

… into filterAndTranslateByReceiver and out of translateByReceiver

Default routing filter is now doesNotMatch(processing_mode_code,T,D).…

0d0c2c7

… Renamed some classes for clarity

jimduff-usds added 11 commits December 1, 2021 12:48

Merge branch 'master' into jim/1244-org-filters

fc16420

sending ReportStreamFilterResults to the next Report

937c6c2

Added test for applyAllFilterTypes

4d4fa63

Merge branch 'master' into jim/1244-org-filters

741cb3d

added testing. Fixed issue in row numbering by using trackingElement …

31e58a7

…instead

commit merge with master

56ae22a

Merge branch 'master' into jim/1244-org-filters

431b290

Changed organizations.yml 'ignore' org to have org-level juris filter.

592f9d8

modified organizations.yml to start using the org-level juris filter …

8eb1a96

…feature, for 'ignore'

function rename cleanup

760aa0f

removed old commented-out code

c415bd7

jimduff-usds commented Dec 5, 2021

View reviewed changes

added AllowNone unit test

3d55e43

jimduff-usds requested review from whytheplatypus, carlosfelix2, MauriceReeves-usds, brick-green, hermanAlexCordero, jbiskie, oslynn and RickHawesUSDS December 6, 2021 02:34

jimduff-usds marked this pull request as ready for review December 6, 2021 02:39

removed spurious debug stuff

0c93243

Removed processing_mode_code filter from qualityFilter and gave it it…

961bf1e

…s own filter

jimduff-usds commented Dec 6, 2021

View reviewed changes

brick-green reviewed Dec 6, 2021

View reviewed changes

Removed old logging code

9934bfb

jimduff-usds requested a review from a team as a code owner December 9, 2021 18:18

jimduff-usds added 4 commits December 9, 2021 15:13

Fixed one unit test to use Carlos' new style

62edfb3

Fixed all my unit tests to follow Carlos' style

bd70478

filter message syntax fix

0644646

Merge branch 'master' into jim/1244-org-filters

37bbfe5

cwinters-usds reviewed Dec 9, 2021

View reviewed changes

cwinters-usds approved these changes Dec 9, 2021

View reviewed changes

Fixing issue causing unit test fails in github

665608b

jimduff-usds commented Dec 9, 2021

View reviewed changes

prime-router/src/main/kotlin/Receiver.kt Outdated Show resolved Hide resolved

jimduff-usds and others added 2 commits December 9, 2021 18:26

fixed comment based on feedback from Brick

4353378

Merge branch 'master' into jim/1244-org-filters

1fd4b0d

jimduff-usds merged commit ad23175 into master Dec 13, 2021

jimduff-usds deleted the jim/1244-org-filters branch December 13, 2021 14:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Jim/1244 org filters #3274

Jim/1244 org filters #3274

jimduff-usds commented Dec 1, 2021 •

edited by github-actions bot

Loading

github-actions bot commented Dec 1, 2021 •

edited

Loading

github-actions bot commented Dec 1, 2021 •

edited

Loading

jimduff-usds Dec 5, 2021

jimduff-usds Dec 5, 2021

jimduff-usds Dec 5, 2021

jimduff-usds Dec 5, 2021 •

edited

Loading

jimduff-usds Dec 5, 2021

jimduff-usds Dec 5, 2021

jimduff-usds Dec 5, 2021

jimduff-usds Dec 5, 2021

jimduff-usds Dec 5, 2021

jimduff-usds Dec 5, 2021

jimduff-usds Dec 6, 2021

brick-green Dec 6, 2021

cwinters-usds Dec 9, 2021

cwinters-usds left a comment

sonarqubecloud bot commented Dec 13, 2021

		private var jurisdictionalFilters = listOf(
		private var reportStreamFilterDefinitions = listOf(

Jim/1244 org filters #3274

Jim/1244 org filters #3274

Conversation

jimduff-usds commented Dec 1, 2021 • edited by github-actions bot Loading

How to Test

Run a smoke test

Print out some quality filter logging

Checklist

Testing

Process

Fixes

To Be Done

Pull reviewers stats

Pull reviewers stats

Pull reviewers stats

Pull reviewers stats

github-actions bot commented Dec 1, 2021 • edited Loading

Unit Test Results

github-actions bot commented Dec 1, 2021 • edited Loading

Integration Test Results

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jimduff-usds Dec 5, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cwinters-usds left a comment

Choose a reason for hiding this comment

sonarqubecloud bot commented Dec 13, 2021

jimduff-usds commented Dec 1, 2021 •

edited by github-actions bot

Loading

github-actions bot commented Dec 1, 2021 •

edited

Loading

github-actions bot commented Dec 1, 2021 •

edited

Loading

jimduff-usds Dec 5, 2021 •

edited

Loading