Mock qa data #6465

Thingus · 2024-02-29T16:18:14Z

Closes #6467

I have:

Formatted any Python files with black
Brought the branch up to date with master
Added any relevant Github labels
Added tests for any new additions
Added or updated any relevant documentation
Added an Architectural Decision Record (ADR), if appropriate
Added an MPLv2 License Header if appropriate
Updated the Changelog

Description

Adds the option to run QA checks on mock data, on by default

To see the specific tasks where the Asana app for GitHub is being used, see below:
- https://app.asana.com/0/0/1206694814086876

cypress · 2024-02-29T16:28:04Z

Passing run #21870 ↗︎

0	4	0	0	0

Details:

Merge branch 'master' into mock_qa_data
Project: FlowAuth	Commit: `720b4a9bc1`
Status: Passed	Duration: 00:42 💡
Started: Mar 18, 2024 4:01 PM	Ended: Mar 18, 2024 4:02 PM

Review all test suite changes for PR #6465 ↗︎

codecov · 2024-02-29T16:36:43Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 92.94%. Comparing base (fb3478b) to head (720b4a9).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6465      +/-   ##
==========================================
- Coverage   93.04%   92.94%   -0.10%     
==========================================
  Files         264      264              
  Lines       10303    10303              
  Branches      835      835              
==========================================
- Hits         9586     9576      -10     
- Misses        589      598       +9     
- Partials      128      129       +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

greenape

Cool.

Can we have an issue number, a changelog entry etc.

I'd also dispute that the qa data is mock - it is real qa about mock data, which is kind of different (i.e. we're actually doing the checks instead of just filling in some made up numbers), so maybe change the names to reflect that.

greenape · 2024-03-07T13:12:49Z

.circleci/config.yml

@@ -35,6 +35,7 @@ defaults:
    FLOWAPI_FLOWDB_USER: flowapi
    FLOWMACHINE_FLOWDB_PASSWORD: foo
    FLOWAPI_FLOWDB_PASSWORD: foo
+    TEST_QA_CHECK: 1


Hm. I kind of feel like this should default to on really.

greenape · 2024-03-07T13:13:07Z

flowdb/testdata/bin/9900_ingest_synthetic_data.sh

@@ -76,3 +76,6 @@ else
    echo "Must set SYNTHETIC_DATA_GENERATOR environment variable to 'sql' or 'python'."
    exit 1
 fi
+if [ "$TEST_QA_CHECK" ]; then
+   pipenv run python ./run_qa_checks.py --dates ${DISASTER_END:-"2015-01-01"} --event-types calls mds sms


Do we not want to just run it for all the dates in the database really?

And all event types

Do we have internal sources of truth for those? I'd guess it's etl.available_dates for dates, but I'm not sure what it would be for the various flavours of events.

etl.etl_records or available_tables

greenape · 2024-03-07T13:15:23Z

flowdb/testdata/bin/run_qa_checks.py

+from itertools import product
+from pathlib import Path
+from typing import List
+from jinja2 import Environment, PackageLoader, Template


Needs adding to the Pipfile and a relock.

greenape · 2024-03-07T13:16:18Z

flowdb/testdata/bin/run_qa_checks.py

+
+
+# NOTE: given this gets run _after_ all the ingestion, 'dates' may be an irrelevence
+# unless final_table is normally one of the dated partitions


The final table will almost always be a dated partition table.

greenape · 2024-03-07T13:17:15Z

flowdb/testdata/bin/run_qa_checks.py

+        for date, type, template in product(qa_scn.dates, qa_scn.tables, templates)
+    )
+
+    with engine.connect() as conn:


iirc you need an explicit commit now, or to use engine.begin().

greenape · 2024-03-07T13:18:42Z

flowdb/testdata/bin/run_qa_checks.py

+    db_port = os.getenv("POSTGRES_PORT", "9000")
+    db_password = os.getenv(
+        "POSTGRES_PASSWORD", "flowflow"
+    )  # In here for dev, don't think we'll need it in prod


You mean for locally running this script I assume?

Use os.environ I think, because you don't esp want it diverging from development_environment.

flowdb/testdata/bin/run_qa_checks.py

greenape · 2024-03-07T13:35:22Z

flowdb/testdata/bin/run_qa_checks.py

+
+env = Environment(loader=PackageLoader("flowetl", "qa_checks/qa_checks"))
+
+# @James; if something like this already exists, I'll grab


It is in the operator, so I think I'd stick to what you have here.

Co-authored-by: Jonathan Gray <[email protected]>

greenape · 2024-03-11T12:16:18Z

flowdb/Pipfile

@@ -8,6 +8,7 @@ psutil = "*"

 [dev-packages]
 black = {extras = ["jupyter"],version = "==24.2.0"}
+jinja2 = "*"


Should go in flowdb/testdata/test_data rather than the root one, isn't needed in the base image, only the test/synth ones.

greenape · 2024-03-11T12:16:57Z

flowdb/testdata/bin/9900_ingest_synthetic_data.sh

@@ -76,3 +76,6 @@ else
    echo "Must set SYNTHETIC_DATA_GENERATOR environment variable to 'sql' or 'python'."
    exit 1
 fi
+if [ "$TEST_QA_CHECK" ]; then
+   pipenv run python ./run_qa_checks.py --dates ${DISASTER_END:-"2015-01-01"} --event-types calls mds sms


etl.etl_records or available_tables

greenape · 2024-03-11T12:18:13Z

flowdb/testdata/bin/run_qa_checks.py

+    db_port = os.getenv("POSTGRES_PORT", "9000")
+    db_password = os.getenv(
+        "POSTGRES_PASSWORD", "flowflow"
+    )  # In here for dev, don't think we'll need it in prod


Use os.environ I think, because you don't esp want it diverging from development_environment.

POSTGRES_DB and POSTGRES_PORT aren't in development_environment; so hardcoded. Not sure if this is a good idea for POSTGRES_PORT.

…into mock_qa_data

greenape · 2024-03-14T10:23:08Z

flowdb/testdata/test_data/Pipfile

@@ -7,6 +7,7 @@ name = "pypi"
 tohu = "*"
 "geoalchemy2" = "*"
 sqlalchemy = "*"
+jinja2 = "*"


I think you'll need to add to testdata/synthetic_data/Pipfile as well

greenape · 2024-03-14T10:24:30Z

flowdb/testdata/bin/run_qa_checks.py

+def get_available_tables(engine: Engine):
+    with engine.begin() as conn:
+        tables = conn.execute(
+            "SELECT table_name FROM available_tables WHERE has_subscribers"


Could also just get this from the available dates one now I think of it, but this is good too :)

flowdb/testdata/bin/run_qa_checks.py

greenape · 2024-03-14T10:30:55Z

flowdb/testdata/bin/run_qa_checks.py

+import argparse
+
+
+env = Environment(loader=PackageLoader("flowetl", "qa_checks/qa_checks"))


Presumably we need to copy these into the container as well?

Just sticking them into the dockerignore should grab it, right?

Co-authored-by: Jonathan Gray <[email protected]>

…into mock_qa_data

greenape

Happy now 👌

Thingus added 5 commits February 29, 2024 09:59

Initial automatic running of all ingestion tests

0ca11ad

Tidying up qa_check script

537296c

Moving qa check script to testdata/bin

fd2fef9

Argparse in run_qa_checks

2e248a6

Adding to ingestion scripts

210fc5f

Adding QA check to circleci to check build

b1b3c46

greenape requested changes Mar 7, 2024

View reviewed changes

Thingus and others added 5 commits March 11, 2024 09:52

Update flowdb/testdata/bin/run_qa_checks.py

ffbccc5

Co-authored-by: Jonathan Gray <[email protected]>

Merge branch 'master' into mock_qa_data

207895c

Comments from review

774b414

Relocking pipfile

2b9e843

CHANGELOG.md

26a53e2

greenape requested changes Mar 11, 2024

View reviewed changes

Thingus added 4 commits March 12, 2024 09:07

Shifting jinja to test pipenv + relocking

8a81141

Comments from PR (wip)

7d549fd

POSTGRES_DB and POSTGRES_PORT aren't in development_environment; so hardcoded. Not sure if this is a good idea for POSTGRES_PORT.

Get default events tables and dates from flowdb

cef5b4d

Merge branch 'master' into mock_qa_data

afc5d7d

Thingus added enhancement New feature or request FlowDB Issues related to FlowDB FlowETL Needs Review PR that is ready for review by a human labels Mar 13, 2024

Thingus marked this pull request as ready for review March 13, 2024 16:05

Thingus requested a review from greenape March 13, 2024 16:05

Thingus added 2 commits March 14, 2024 10:17

Moving run_qa_checks to bin + adding to dockerignoreignore

a23b9f0

Merge branch 'mock_qa_data' of https://github.com/Flowminder/FlowKit …

289ce8b

…into mock_qa_data

greenape reviewed Mar 14, 2024

View reviewed changes

flowdb/testdata/bin/run_qa_checks.py Outdated Show resolved Hide resolved

greenape reviewed Mar 14, 2024

View reviewed changes

Thingus and others added 20 commits March 14, 2024 10:48

Update flowdb/testdata/bin/run_qa_checks.py

bf1d1e2

Co-authored-by: Jonathan Gray <[email protected]>

Jinja2 and relocking for synthdata

91b78ae

Merge branch 'mock_qa_data' of https://github.com/Flowminder/FlowKit …

2cf85a5

…into mock_qa_data

Adding flowetl qa templates to flowdb test data containers

46eb93e

Maybe its those commas causing trouble

470335d

Now using pop and pushdir in test_data scripts

ae01aaf

Some container messing

6f074b5

pushd not pushdir

8349130

Pipenv messing

9f4b8fa

Ah, the pipfile is in the root

09cac3c

Lets try an explicity path

c10739a

run_qa_checks now runs on local container

57ca678

More prints + popenv run

0b265e1

cd instead of push/popd

4e837a9

Minor fixes

8bd4223

Explcit copy of run_qa_checks.py for synth data

2b7c409

Adding run_qa_checks.py to synth data image

9cf2611

Merge branch 'mock_qa_data' of https://github.com/Flowminder/FlowKit …

15d5928

…into mock_qa_data

Removing extra COPY command

a346ddd

Merge branch 'master' into mock_qa_data

720b4a9

greenape approved these changes Mar 18, 2024

View reviewed changes

Thingus merged commit ba9948f into master Mar 19, 2024
39 of 40 checks passed

Thingus deleted the mock_qa_data branch March 19, 2024 09:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mock qa data #6465

Mock qa data #6465

Thingus commented Feb 29, 2024 •

edited

Loading

cypress bot commented Feb 29, 2024 •

edited

Loading

codecov bot commented Feb 29, 2024 •

edited

Loading

greenape left a comment

greenape Mar 7, 2024

greenape Mar 7, 2024

greenape Mar 7, 2024

Thingus Mar 11, 2024

greenape Mar 11, 2024

greenape Mar 7, 2024

greenape Mar 7, 2024

greenape Mar 7, 2024

greenape Mar 7, 2024

Thingus Mar 11, 2024

greenape Mar 11, 2024

greenape Mar 7, 2024

greenape Mar 11, 2024

greenape Mar 11, 2024

greenape Mar 11, 2024

greenape Mar 14, 2024

greenape Mar 14, 2024

greenape Mar 14, 2024

Thingus Mar 14, 2024

greenape left a comment



		# NOTE: given this gets run _after_ all the ingestion, 'dates' may be an irrelevence
		# unless final_table is normally one of the dated partitions


		env = Environment(loader=PackageLoader("flowetl", "qa_checks/qa_checks"))

		# @James; if something like this already exists, I'll grab

		import argparse


		env = Environment(loader=PackageLoader("flowetl", "qa_checks/qa_checks"))

Mock qa data #6465

Mock qa data #6465

Conversation

Thingus commented Feb 29, 2024 • edited Loading

I have:

Description

cypress bot commented Feb 29, 2024 • edited Loading

Passing run #21870 ↗︎

Review all test suite changes for PR #6465 ↗︎

codecov bot commented Feb 29, 2024 • edited Loading

Codecov Report

greenape left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

greenape left a comment

Choose a reason for hiding this comment

Thingus commented Feb 29, 2024 •

edited

Loading

cypress bot commented Feb 29, 2024 •

edited

Loading

codecov bot commented Feb 29, 2024 •

edited

Loading