Implement output schema #51
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
stevebachmeier
requested review from
albrja,
hussain-jafari,
mattkappel,
ramittal and
rmudambi
as code owners
mattkappel
reviewed
Apr 11, 2023
mattkappel
approved these changes
Apr 11, 2023
rmudambi
reviewed
Apr 11, 2023
There was a problem hiding this comment.
Out of scope for this PR, but this almost makes incorrect_select_options.csv unnecessary.
stevebachmeier
commented
Apr 12, 2023
albrja
approved these changes
Apr 12, 2023
rmudambi
approved these changes
Apr 12, 2023
| @@ -389,6 +389,9 @@ def test_miswrite_numerics(string_series): | |||
| p_row_noise = config.row_noise_level | |||
| p_token_noise = config.token_noise_level | |||
| data = string_series | |||
| # Hack: we need to name the series something with the miswrite_numeric noising | |||
| # function applied to check dtypes. | |||
| data.name = "ssn" | |||
There was a problem hiding this comment.
Why not actually give our dummy_data appropriate column names when we define the fixture?
There was a problem hiding this comment.
That's just an old fixture and didn't seem worth doing atm.
…/sbachmei/MIC-3961-subset-output-columns
…/sbachmei/MIC-3961-subset-output-columns
stevebachmeier
deleted the
feature/sbachmei/MIC-3961-subset-output-columns
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Title: Implement output schema (columns and dtypes)
Description
The primary goal of this is to have the noising function output specific columns
rather than all columns required for noising. It should also output the columns
in the correct order.
A secondary goal is to start getting a handle on dtypes. This may not be necessary
anymore, however, now that we have moved away from csv files and to hdf and
parquet (both of which store dtype data).
NOTES:
error very late in noising when in reality the users likely won't care about dtypes
Is it even worth managing dtypes anymore?
Testing
pytests pass.
TODO: possibly create a dtype pytest if it's decided we want to
keep that functionality.