merge main into configurableDelimiterSeeds #1
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Update semantic model parsing test to check different measure expr types * Allow semantic model measure exprs to be defined with ints and bools in yaml Sometimes the expr for a measure can defined in yaml with a bool or an int. However, we were only allowing for strings. There was a work around for this, which was wrapping your bool or int in double quotes in the yaml, but this can be fairly annoying for the end user. * Changie doc for fixing measure expr yaml specification
* Update semantic model parsing test to check measure agg params * Make `use_discrete_percentile` and `use_approximate_percentile` non optional and default false This was a mistake in our implementation of the MeasureAggregationParams. We had defined them as optional and defaulting to `None`. However, as the protocol states, they cannot be `None`, they must be a boolean value. Thus now we now ensure them. * Add changie doc for measure percentile fixes
…uring release process
* Update mashumaro to 3.8 * Change to 3.8.1 * Changie
* CT-2711: Add remove_tests() call to delete_schema_source() so that call sites are more uniform with other node deletion call sites. This will enable further code factorization. * CT-2711: Factor repeated code section (mostly) out of PartialParsing.handle_schema_file_changes() * CT-2711: Factor a repeated code section out of schedule_nodes_for_parsing()
* CT-2711: Add partial parsing support for semantic models * CT-2711: Fix typo identified in code review
* Add merge incremental strategy * Expect merge to be a valid strategy for Postgres --------- Co-authored-by: Anders Swanson <[email protected]> Co-authored-by: Doug Beatty <[email protected]>
* Add thread_id context var * Changie * Fix context test * Update .changes/unreleased/Features-20230623-173357.yaml Co-authored-by: Doug Beatty <[email protected]> * Fix tests --------- Co-authored-by: Doug Beatty <[email protected]>
* Add invocation command flag * Add changie entry * Update .changes/unreleased/Features-20230623-111254.yaml
…uring release process
…textVar (#7949) * Add task contextvars to events/contextvars.py * Use events.contextvars instead of task.contextvars * Changie
* Fix tests fixtures which were using measures for metric numerator/denominators In our previous upgrade to DSI dev7, numerators and denominators for metrics switched from being `MetricInputMeasure`s to `MetricInput`s. I.e. metric numerators and denominators should references other metrics, not semantic model measures. However, at that time, we weren't actually doing anything with numerators and denominators in core, so no issue got raised. The changes we are about to make though are going to surface these issues.. * Add tests for ensuring a metric's `input_measures` gets properly populated * Begin populating `metric.type_params.input_measures` This isn't my favorite bit of code. Mostly because there are checks for existence which really should be handled before this point, however a good point for that to happen doesn't exist currently. For instance, in an ideal world by the time we get to `_process_metric_node`, if a metric is of type `RATIO` and the nominator and denominator should be guaranteed. * Update test checking that disabled metrics aren't added to the manifest metrics We updated from the metric `number_of_people` to `average_tenure_minus_people` for this test because disabling `number_of_people` raised other exceptions at parse time due to a metric referencing a disabled metric. The metric `average_tenure_minus_people` is a leaf metric, and so for this test, it is a better candidate. * Update `test_disabled_metric_ref_model` to have more disabled metrics There are metrics which depend on the metric `number_of_people`. If `number_of_people` is disabled without the metrics that depend on it being disabled, then a different (expected) exception would be raised than the one this test is testing for. Thus we've disabled those downstream metrics. * Add test which checks that metrics depending on disabled metrics raise an exception * Add changie doc for populating metric input measures
* UnifiedToUTC * Check proximity of dbt_valid_to and deleted time * update the message to print if the assertion fails * add CHANGELOG entries * test only if naive * Added comments about naive and aware * Generalize comparison of datetimes that are "close enough" --------- Co-authored-by: Doug Beatty <[email protected]>
* ct-2551: adds old and unmodified state selection methods * ct-2551: update check_unmodified_content to simplify * add unit and integration tests for unmodified and old * add changelog entry * ct-2551: reformatting of contingent adapter assignment list
…uring release process
…apsulation of details. (#8012)
* docs(contributing): fix-toc-link-in-contributing-md * docs(contributing-md): fix-link2 * Fix-typo * Remove backtick from href * changie new * Cough commit / trigger CI
* pin click * changelog
* add workflow for flaky test testing * improve docs * rename workflow * update default input * add min passing tests
* add interface changes section to the PR template * update entire template * split up choices for tests and interfaces * minor formatting change * add line breaks * actually put in line breaks * revert split choices in checklist * add line breaks to top * move docs link * typo
* Use dbt-semantic-interface validations on semantic models and metrics defined in Core. * Remove empty test, since semantic models don't generate any validation warnings. * Add changelog entry. * Temporarily remove requirement that there must be semantic models definied in order to define metrics
* Fix accidental propagation of log messages to root logger. * Add changelog entry * Fixed an issue which blocked debug logging to stdout with --log-level debug, unless --debug was also used.
* add negative test case * changie * missed a comma * Update changelog entry * Add a negative number (rather than subtract a positive number) --------- Co-authored-by: Doug Beatty <[email protected]> Co-authored-by: Doug Beatty <[email protected]>
…perty (#8068) * Remove `create_metric` as a public facing `SemanticModel.Measure` property We want to add `create_metric`. The `create_metric` property will be incredibly useful. However, at this time it is not hooked up, and we don't have time to hook it up before the code freeze for 1.6.0rc of core. As it doesn't do anything, we shouldn't allow people to specify it, because it won't do what one would expect. We plan on making the implementation of `create_metric` a priority for 1.7 of core * Changie doc for the removal of create_metric property
* add permissions * replace db setup * try with bash instead of just pytest flags * fix test command * remove spaces * remove force-flaky flag * add starting vlaues * add mac and windows postgres isntall * define use bash * fix typo * update output report * tweak last if condition * clarify failures/successful runs * print running success and failure tally * just output pytest instead of capturing it * set shell to not exit immediately on exit code * add formatting around results for easier scanning * more output formatting * add matrix to unlock parallel runners * increase to ten batches * update debug * add comment * clean up comments
…orted release (#8071) * Use Ubuntu 22.04.2 LTS (Jammy Jellyfish) since it is a long-term supported release * Changelog entry
…apters (#8001) * Add dry_run method to base adapter with implementation for SQLAdapters resolves #7839 In the CLI integration, MetricFlow will issue dry run queries as part of its warehouse-level validation of the semantic manifest, including all semantic model and metric definitions. In most cases, issuing an `explain` query is adequate, however, BigQuery does not support the `explain` keyword and so we cannot simply pre-pend `explain` to our input queries and expect the correct behavior across all contexts. This commit adds a dry_run() method to the BaseAdapter which mirrors the execute() method in that it simply delegates to the ConnectionManager. It also adds a working implementation to the SQLConnectionManager and includes a few test cases for adapter maintainers to try out on their own. The current implementation should work out of the box with most of our adapters. BigQuery will require us to implement the dry_run method on the BigQueryConnectionManager, and community-maintained adapters can opt in by enabling the test and ensuring their own implementations work as expected. Note - we decided to make these concrete methods that throw runtime exceptions for direct descendants of BaseAdapter in order to avoid forcing community adapter maintainers to implement a method that does not currently have any use cases in dbt proper. * Switch dry_run implementation to be macro-based The common pattern for engine-specific SQL statement construction in dbt is to provide a default macro which can then be overridden on a per-adapter basis by either adapter maintainers or end users. The advantage of this is users can take advantage of alternative SQL syntax for performance or other reasons, or even to enable local usage if an engine relies on a non-standard expression and the adapter maintainer has not updated the package. Although there are some risks here they are minimal, and the benefit of added expressiveness and consistency with other similar constructs is clear, so we adopt this approach here. * Improve error message for InvalidConnectionError in test_invalid_dry_run. * Rename dry_run to validate_sql The validate_sql name has less chance of colliding with dbt's command nomenclature, both now and in some future where we have dry-run operations. * Rename macro and test files to validate_sql * Fix changelog entry
The original implementation of validate_sql was called dry_run, but in the rename the test classes and much of their associated documentation still retained the old naming. This is mainly cosmetic, but since these test classes will be imported into adapter repositories we should fix this now before the wrong name proliferates.
* Remove volume declaration * Changelog entry --------- Co-authored-by: Doug Beatty <[email protected]>
* remove skip * fix retry test
* Add semantic_models to tracked resource counts * Add changelog entry * Simplify node statistic tabulation. * Remove review comment. Replace with explanation. --------- Co-authored-by: Peter Allen Webb <[email protected]>
* Add metrics from metric type params to a metric's depends_on * Add Lookup utility for finding `SemanticModel`s by measure names * Add the `SemanticModel` of a `Metric`'s measure property to the `Metric`'s `depends_on` * Add `SemanticModelConfig` to `SemanticModel` Some tests were failing due to `Metric`'s referencing `SemanticModels`. Specifically there was a check to see if a referenced node was disabled, and because `SemanticModel`'s didn't have a `config` holding the `enabled` boolean attr, core would blow up. * Checkpoint on test fixing * Correct metricflow_time_spine_sql in test fixtures * Add check for `SemanticModel` nodes in `Linker.link_node` Now that `Metrics` depend on `SemanticModels` and `SemanticModels` have their own dependencies on `Models` they need to be checked for in the `Linker.link_node`. I forget the details but things blow up without it. Basically it adds the SemanticModels to the dependency graph. * Fix artifacts/test_previous_version_state.py tests * fix access/test_access.py tests * Fix function metric tests * Fix functional partial_parsing tests * Add time dimension to semantic model in exposures fixture * Bump DSI version to a minimum of 0.1.0dev10 DSI 0.1.0dev10 fixes an incoherence issue in DSI around `agg_time_dimension` setting. This incoherence was that `measure.agg_time_dimension` was being required, even though it was no longer supposed to be a required attribute (it's specificially typed as optional in the protocol). This was causing a handful of tests to fail because the `semantic_model.defaults.agg_time_dimension` value wasn't being respected. Pulling in the fix from DSI 0.1.0dev10 fixes the issue. Interestingly after bumping the DSI version, the integration tests were still failing. If I ran the tests individually they passed though. To get `make integration` to run properly I ended up having to clear my `.tox` cache, as it seems some outdated state was being persisted. * Add test specifically for checking the `depends_on` of `Metric` nodes * Re-enable test asserting calling metric nodes in models * Migrate `checked_agg_time_dimension` to `checked_agg_time_dimension_for_measure` DSI 0.1.0dev10 moved `checked_agg_time_dimension` from the `Measure` protocol to the `SemanticModel` protocol as `checked_agg_time_dimension_for_measure`. This finishes a change where for a given measure either the `Measure.agg_time_dimension` or the measure's parent `SemanticModel.defaults.agg_time_dimension` needs to be set, instead of always require the measure's `Measure.agg_time_dimension`. * Add changie doc for populating metric --------- Co-authored-by: Gerda Shank <[email protected]>
* Bump version support for `dbt-semantic-interfaces` to `~=0.1.0rc1` * Add tests for asserting WhereFilter satisfies protocol * Add `call_parameter_sets` to `WhereFilter` class to satisfy protocol * Changie doc for moving to DSI 0.1.0rc1 * [CT-2822] Fix `NonAdditiveDimension` Implementation (#8089) * Add test to ensure `NonAdditiveDimension` implementation satisfies protocol * Fix typo in `NonAdditiveDimension`: `window_grouples` -> `window_groupings` * Add changie doc for typo fix in NonAdditiveDimension
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
resolves #
Description
Checklist
changie newto create a changelog entry