DOC: Reorder noise functions in user docs #254
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
NathanielBlairStahn
added
the
documentation
| @@ -110,60 +110,60 @@ Noise types for each column | |||
| :header-rows: 1 | |||
|
|
|||
| * - Column name | |||
| - Datasets present | |||
| - Applicable datasets | |||
There was a problem hiding this comment.
I went ahead and changed this column name as discussed at the doc session last week. This is unrelated to the noise function order.
| - Leave a field blank, use a fake name, use a nickname, make typos, make OCR errors, make phonetic errors | ||
| - | ||
| - Leave a field blank, use a nickname, use a fake name, make phonetic errors, make OCR errors, make typos | ||
| - Middle names use the same lists of nicknames and fake names used for first names |
There was a problem hiding this comment.
I thought it would be good to specify that middle names use the same data as first names for these two noise functions. This is unrelated to the noise function order.
| - Leave a field blank, use a fake name, make typos, make OCR errors, make phonetic errors | ||
| - In the 1040 form, the same noise types apply to the last name columns for the joint filer and dependents | ||
| - Leave a field blank, use a fake name, make phonetic errors, make OCR errors, make typos | ||
| - Last names use a different list of fake names than the list for first names. In the 1040 form, the same noise types apply to the last name columns for the joint filer and dependents |
There was a problem hiding this comment.
Likewise, I thought it would be good to clarify that last names use a different fake name list than first names. This is unrelated to the noise function order.
NathanielBlairStahn
requested review from
zmbc,
pletale and
a team
as code owners
zmbc
approved these changes
Aug 17, 2023
pletale
approved these changes
Aug 18, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Title: Re-order the noise functions on the noise page and column noise page
Description
Edits the user docs to list the noise functions in the order in which the're applied in the code. Also makes a couple unrelated minor edits to one of the noise tables (see code comments below).
Technically, I put the noise types in an order that should be equivalent to the order in which they're actually applied, but which I think will be more intuitive to a user reading the docs. I would like feedback on whether the order here looks good or whether we want to switch anything. If this ordering looks good, I'll edit the concept model accordingly and create an engineering ticket to update the code to match.
In particular, I have the following questions about what is the most logical order to try to match the actual data generation process:
I think it makes sense that OCR errors and typos are the last two types of noise, but I don't know which of these should come first. Is it more likely that a document is read in by OCR, and then someone types the ORC'ed data into an administrative file, or is it more likely that someone types data into a file, the file is printed, and then the printed file is read in by OCR and input directly into the administrative file?
I currently have nicknames applied before fake names. Does this make sense, or would it make more sense to apply fake names and then nicknames? (In practice, it probably doesn't make much difference because there are probably very few, if any, fake names that are also eligible for nicknames.)
Testing
Built docs locally.