Skip to content

Commit

Permalink
[minor] made a quick fix to the notebook and also added credit to the…
Browse files Browse the repository at this point in the history
… contributors to the readme.
  • Loading branch information
aangelopoulos committed Dec 22, 2024
1 parent cbe850c commit 5b801f0
Show file tree
Hide file tree
Showing 3 changed files with 19 additions and 14 deletions.
3 changes: 3 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,7 @@ There is also a file, ```./ppi_py/baselines.py```, which implements several base
Finally, the file ```./ppi_py/datasets/datasets.py``` handles the loading of the sample datasets.

The folder ```./examples``` contains notebooks for implementing prediction-powered inference on several datasets and estimands. These are listed [above](https://github.com/aangelopoulos/ppi_py/tree/main#examples). There is also an additional subfolder, ```./examples/baselines```, which contains comparisons to certain baseline algorithms, as in the appendix of the original PPI paper.
There is an additional notebook, [```./examples/ppi_power_analysis.py```](https://github.com/aangelopoulos/ppi_py/blob/main/examples/power_analysis.ipynb), which shows how to choose the optimal labeled and unlabeled dataset sizes subject to a constraint on the budget.

The folder ```./tests``` contains unit tests for each function implemented in the ```ppi_py``` package. The tests are organized by estimand, and can be run by executing ```pytest``` in the root directory. Some of the tests are stochastic, and therefore, have some failure probability, even if the functions are all implemented correctly. If a test fails, it may be worth running it again. Debugging the tests can be done by adding the ```-s``` flag and using print statements or ```pdb```. Note that in order to be recognized by ```pytest```, all tests must be preceded by ```test_```.

Expand Down Expand Up @@ -141,3 +142,5 @@ The repository currently implements the methods developed in the following paper
[Cross-Prediction-Powered Inference](https://arxiv.org/abs/2309.16598)

[Prediction-Powered Bootstrap](https://arxiv.org/abs/2405.18379)

[The Mixed Subjects Design: Treating Large Language Models as (Potentially) Informative Observations](https://osf.io/preprints/socarxiv/j3bnt)
2 changes: 2 additions & 0 deletions examples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,3 +16,5 @@ Each notebook runs a simulation that forms a dataframe containing confidence int
- the average interval width for PPI and the classical method, together with a scatterplot of the widths from the five random draws.

Each notebook also compares PPI and classical inference in terms of the number of labeled examples needed to reject a natural null hypothesis in the analyzed problem.

Finally, there is a notebook that shows how to compute the optimal `n` and `N` given a cost constraint ([```power_analysis.ipynb```](https://github.com/aangelopoulos/ppi_py/blob/main/examples/power_analysis.ipynb)).
28 changes: 14 additions & 14 deletions examples/power_analysis.ipynb

Large diffs are not rendered by default.

0 comments on commit 5b801f0

Please sign in to comment.