Validation updates to better support UTMs and regex rules #598

NickPhura · 2021-10-15T22:49:16Z

Overview

https://quartech.atlassian.net/browse/BHBC-1428
https://quartech.atlassian.net/browse/BHBC-1460

This PR contains the following changes

Validation/Transformation

Update format validation schema to support regex flags
- Update validation schemas to have updated rules for waypoint and utm zone
- Update spatial-utils to allow UTM zone letters to be optional
Update transformation/validation schemas for both moose templates
- Recruitment should now work
Update XLSX transformation logic to support better support conditional rules
- This should also fix BHBC-1460

N8N

Pin n8n version to 0.144.0
Update n8n workflow exports (latest n8n version has modified the config)

Scraping Occurrences

Update occurrence table to add sex column
Update occurrence popup to include the sex field
Tweak occurrence popup to better handle occurrences with no geometry

This PR contains the following types of changes

New feature (change which adds functionality)
Enhancement (improvements to existing functionality)
Bug fix (change which fixes an issue)
Misc cleanup / Refactoring / Documentation
Version change

Checklist

A list of items that are good to consider when making any changes.

Note: this list is not exhaustive, and not all items are always applicable.

General

I have performed a self-review of my own code

Code

New files/classes/functions have appropriately descriptive names and comment blocks to describe their use/behaviour
I have avoided duplicating code when possible, moving re-usable pieces into functions
I have avoided hard-coding values where possible and moved any re-usable constants to a constants file
My code is as flat as possible (avoids deeply nested if/else blocks, promise chains, etc)
My code changes account for null/undefined values and handle errors appropriately
My code uses types/interfaces to help describe values/parameters/etc, help ensure type safety, and improve readability

Style

My code follows the established style conventions
My code uses native material-ui components/icons/conventions when possible

Documentation

I have commented my code sufficiently, such that an unfamiliar developer could understand my code
I have added/updated README's and related documentation, as needed

Tests

I have added/updated unit tests for any code I've added/updated
I have added/updated the Postman requests/tests to account for any API endpoints I've added/updated

Linting/Formatting

I have run the linter and fixed any issues, as needed
See the lint commands in package.json
I have run the formatter and fixed any issues, as needed
See the format commands in package.json

SonarCloud

I have addressed all SonarCloud Bugs, Vulnerabilities, Security Hotspots, and Code Smells

How Has This Been Tested?

Please describe the tests that you ran to verify your changes.

Screenshots

Please add any relevant UI screenshots, if applicable.

- Update spatial-utils to allow UTM zone letters to be optional - Update validation schemas to have updated rules for waypoint and utm zone - Pin n8n version to 0.144.0 - Update n8n workflow exports (latest n8n version has modified the config)

codecov · 2021-10-15T22:50:56Z

Codecov Report

Merging #598 (c69aea8) into dev (038d506) will decrease coverage by 0.10%.
The diff coverage is 21.05%.

@@            Coverage Diff             @@
##              dev     #598      +/-   ##
==========================================
- Coverage   76.80%   76.70%   -0.11%     
==========================================
  Files         287      287              
  Lines       10250    10281      +31     
  Branches     2141     2150       +9     
==========================================
+ Hits         7873     7886      +13     
- Misses       2043     2063      +20     
+ Partials      334      332       -2

Impacted Files	Coverage Δ
api/src/json-schema/validation-schema.ts	`100.00% <ø> (ø)`
api/src/models/occurrence-view.ts	`60.00% <ø> (ø)`
api/src/paths/dwc/scrape-occurrences.ts	`0.00% <0.00%> (ø)`
...rc/queries/occurrence/occurrence-create-queries.ts	`100.00% <ø> (ø)`
.../src/queries/occurrence/occurrence-view-queries.ts	`100.00% <ø> (ø)`
...lsx/transformation/transformation-schema-parser.ts	`43.47% <0.00%> (ø)`
app/src/components/map/OccurrenceFeatureGroup.tsx	`5.00% <0.00%> (-0.27%)`	⬇️
app/src/components/map/OccurrenceFeaturePopup.tsx	`25.00% <ø> (ø)`
...s/media/xlsx/transformation/xlsx-transformation.ts	`9.23% <3.84%> (-0.03%)`	⬇️
api/src/models/occurrence-create.ts	`100.00% <100.00%> (ø)`
... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 038d506...c69aea8. Read the comment docs.

….com/bcgov/biohubbc into update_utm_validation_transformation

rstens · 2021-10-19T00:11:46Z

Sonarqube is griping about the regex - it states that you should use \d instead of [0-9] or [1-2]? instead of [1-2]{0,1}

anissa-agahchen · 2021-10-19T15:20:53Z

app/src/components/map/OccurrenceFeatureGroup.tsx

+          return (
+            <Marker
+              key={occurrence.occurrenceId}
+              position={[(geometry.geometry as Point).coordinates[1], (geometry.geometry as Point).coordinates[0]]}>


Suggested change

position={[(geometry.geometry as Point).coordinates[1], (geometry.geometry as Point).coordinates[0]]}>

position={[(geometry?.geometry as Point).coordinates[1], (geometry?.geometry as Point).coordinates[0]]}>

Nit: spelling of this file should be

database/src/migrations/20211018010922_update_funding_source.ts

anissa-agahchen · 2021-10-19T15:48:44Z

api/src/utils/spatial-utils.ts

@@ -12,7 +12,8 @@ const SOPUTH_UTM_BASE_ZONE_NUMBER = 32700;
 const NORTH_UTM_ZONE_LETTERS = ['N', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X'];
 const SOUTH_UTM_ZONE_LETTERS = ['C', 'D', 'E', 'F', 'G', 'H', 'J', 'K', 'L', 'M'];

-const UTM_STRING_FORMAT = RegExp(/^[1-9]{0,1}[0-9][NPQRSTUVWXCDEFGHJKLM] [0-9]+ [0-9]+$/i);
+const UTM_STRING_FORMAT = RegExp(/^[1-9]\d?[NPQRSTUVWXCDEFGHJKLM]? \d+ \d+$/i);
+const UTM_ZONE_WITH_LETTER_FORMAT = RegExp(/^[1-9]\d?[NPQRSTUVWXCDEFGHJKLM]$/i);


NickPhura · 2021-10-19T16:24:32Z

Sonarqube is griping about the regex - it states that you should use \d instead of [0-9] or [1-2]? instead of [1-2]{0,1}

I've made some changes, so it should be happy now

Fix occurrence feature group logic.

….com/bcgov/biohubbc into update_utm_validation_transformation

sonarqubecloud · 2021-10-19T19:07:00Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
2.3% Duplication

NickPhura requested a review from anissa-agahchen October 15, 2021 22:49

NickPhura and others added 2 commits October 15, 2021 15:49

Update test wording

a0a66ab

Merge branch 'dev' into update_utm_validation_transformation

14bfdab

NickPhura added 4 commits October 15, 2021 15:56

Update utm regex

395ac9d

Merge branch 'update_utm_validation_transformation' of https://github…

43e492a

….com/bcgov/biohubbc into update_utm_validation_transformation

Undue validation schema changes for old version

8d4736b

Fix bug in parse schema

d880864

NickPhura added 4 commits October 18, 2021 18:26

Updates

4a66d89

Updates

543ab35

updates

0b8ba3f

Updates

866dd1d

anissa-agahchen reviewed Oct 19, 2021

View reviewed changes

NickPhura added 2 commits October 19, 2021 09:22

Update geometry marker group

1611c53

Fix migration file name

deeaef1

NickPhura added 2 commits October 19, 2021 09:33

Remove png from report type

928c135

Handle transformation IDs as lowercase.

1f28f8e

Fix occurrence feature group logic.

anissa-agahchen previously approved these changes Oct 19, 2021

View reviewed changes

NickPhura and others added 3 commits October 19, 2021 11:13

Merge branch 'dev' into update_utm_validation_transformation

9185e99

Delete occurrence sex migration

8a3f12d

Merge branch 'update_utm_validation_transformation' of https://github…

89f6ceb

….com/bcgov/biohubbc into update_utm_validation_transformation

NickPhura dismissed anissa-agahchen’s stale review via 89f6ceb October 19, 2021 18:14

NickPhura added 3 commits October 19, 2021 11:39

Merge branch 'dev' into update_utm_validation_transformation

e652b1e

Merge branch 'dev' into update_utm_validation_transformation

1d3f302

Merge branch 'dev' into update_utm_validation_transformation

c69aea8

anissa-agahchen approved these changes Oct 19, 2021

View reviewed changes

NickPhura merged commit b74c763 into dev Oct 19, 2021

NickPhura deleted the update_utm_validation_transformation branch October 19, 2021 19:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Validation updates to better support UTMs and regex rules #598

Validation updates to better support UTMs and regex rules #598

NickPhura commented Oct 15, 2021 •

edited

Loading

codecov bot commented Oct 15, 2021 •

edited

Loading

rstens commented Oct 19, 2021

anissa-agahchen Oct 19, 2021

anissa-agahchen Oct 19, 2021

NickPhura Oct 19, 2021

anissa-agahchen Oct 19, 2021

NickPhura commented Oct 19, 2021

sonarqubecloud bot commented Oct 19, 2021

	position={[(geometry.geometry as Point).coordinates[1], (geometry.geometry as Point).coordinates[0]]}>
	position={[(geometry?.geometry as Point).coordinates[1], (geometry?.geometry as Point).coordinates[0]]}>

Validation updates to better support UTMs and regex rules #598

Validation updates to better support UTMs and regex rules #598

Conversation

NickPhura commented Oct 15, 2021 • edited Loading

Overview

This PR contains the following changes

Validation/Transformation

N8N

Scraping Occurrences

This PR contains the following types of changes

Checklist

General

Code

Style

Documentation

Tests

Linting/Formatting

SonarCloud

How Has This Been Tested?

Screenshots

codecov bot commented Oct 15, 2021 • edited Loading

Codecov Report

rstens commented Oct 19, 2021

anissa-agahchen Oct 19, 2021

Choose a reason for hiding this comment

anissa-agahchen Oct 19, 2021

Choose a reason for hiding this comment

NickPhura Oct 19, 2021

Choose a reason for hiding this comment

anissa-agahchen Oct 19, 2021

Choose a reason for hiding this comment

NickPhura commented Oct 19, 2021

sonarqubecloud bot commented Oct 19, 2021

NickPhura commented Oct 15, 2021 •

edited

Loading

codecov bot commented Oct 15, 2021 •

edited

Loading