Add logging and progress bar for generation of forms

[mattkappel]

Add logging and progress bar for generation of forms

Description

• Category: feature
• JIRA issue: MIC-3987

Changes

• Adds tqdm to setup requirements
• Adds tqdm progress bar over the main noise_form loop
• Adds a logging message on opening a data file
• Sets default log level to INFO, configurable to DEBUG if verbose=True on an interface

Testing

Interactive output looks as expected:

>>> import pseudopeople as pp
>>> data = pp.generate_decennial_census()
2023-04-14 10:16:37.443 | INFO     | pseudopeople.interface:_generate_form:62 - Loading data from /Users/mkappel/git/pseudopeople/pseudopeople/src/pseudopeople/data/sample_forms/decennial_census_observer/decennial_census_observer.parquet.
Applying noise: 100%|███████████████████████████| 8/8 [00:00<00:00, 53.98type/s]
>>> data2 = pp.generate_taxes_w2_and_1099()
2023-04-14 10:19:22.760 | INFO     | pseudopeople.interface:_generate_form:62 - Loading data from /Users/mkappel/git/pseudopeople/pseudopeople/src/pseudopeople/data/sample_forms/tax_w2_observer/tax_w2_observer.parquet.
Applying noise: 100%|███████████████████████████| 8/8 [00:00<00:00, 25.11type/s]

[stevebachmeier]

I love tqdm - it's just so easy. But what is unit="type" doing?

[stevebachmeier]

Oh, it is doing xxx type/s. Hmm...

[stevebachmeier]

maybe unit="noise types"?

[mattkappel]

The default it "it" so instead of "it/sec" it says "type/sec". I mean "type" to be noise type, but it could be any string we want.

[mattkappel]

I wanted to keep it short to save on the terminal space. There is already the description of "Applying noise". There should be no ambiguity there...

[rmudambi]

Two notes:

• We should set the default log level to INFO if it is not already at that level. That should be part of this PR
• How does this work when there are multiple files being noised (i.e. say there were two decennial census files in the sample data directory rather than 1)? When they run on the full data they will have 334 files. This does not need to be resolved in this PR.

[stevebachmeier]

• How does this work when there are multiple files being noised (i.e. say there were two decennial census files in the sample data directory rather than 1)? When they run on the full data they will have 334 files. This does not need to be resolved in this PR.

tqdm is currently wrapping the actual noising loop, so each of the 334 files would get their own progress bar one after another since it's done in serial.

[mattkappel]

@rmudambi What Steve said. I tried wrapping the file reading with tqdm, too, but that got messy, since the loops are intertwined.

I'm still looking into where/how we ought to set the log level.