Bulk Upload Fhir Conversion

[DanielSass]

BACKEND PULL REQUEST

Related Issue

• Convert CSV schema to FHIR #5069

Changes Proposed

• Add functionality to create FHIR Bundles from the Test Result Bulk Upload csv"

Testing

I did some manual testing with very large files just to verify how things behave. I did this with repeated records (5 records copied and pasted until we had x# in the file) so it's not particularly prod-like. Might look at writing a quick python script to generate data with faker or something like that in the future.

For context: 9999 records is the most the front end will let you submit. I believe looking at metabase the largest file uploads we've seen in prod are ~4000 records. The overwhelming majority are under ~2000 records.

Of note - these timings don't include the report stream interaction, and for larger files I've noticed this is significant - at least for local dev.

Execution time pre-async

Execution time for conversion: 9325 milliseconds
2225 Records converted.
Execution time for parse: 1193 milliseconds

Execution time for conversion: 23478 milliseconds
5000 Records converted.
Execution time for parse: 2703 milliseconds

Execution time for conversion: 50486 milliseconds
9999 Records converted.
Execution time for parse: 5427 milliseconds

Execution time post-async

Execution time for conversion & parse: 1708 milliseconds
2500 Records converted.

Execution time for conversion & parse: 1920 milliseconds
5000 Records converted.

Execution time for conversion & parse: 3790 milliseconds
9998 Records converted.

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
1 Code Smell

82.4% Coverage
0.0% Duplication

[zdeveloper]

LGTM! thank you for putting this together, would love to see a test that has a csv input file and an expected bundle.ndjson file (could be added in future PR since this is a foundational pr)

[BobanL]

Looks great! Thanks 😃