Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Bugfix/numerframe meta property #118

Merged
merged 3 commits into from
Nov 18, 2022
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 36 additions & 21 deletions README.md
Original file line number Diff line number Diff line change
@@ -8,8 +8,9 @@ about software/data engineering and focus more on building great Numerai
models!

Most of the components in this library are designed for solid weekly
inference pipelines, but tools like `NumerFrame`, preprocessors and
evaluators also greatly simplify the training process.
inference pipelines, but tools like
[`NumerFrame`](https://crowdcent.github.io/numerblox/numerframe.html#numerframe),
preprocessors and evaluators also greatly simplify the training process.

**Questions and discussion:**
[rocketchat.numer.ai/channel/numerblox](https://rocketchat.numer.ai/channel/numerblox)
@@ -63,36 +64,50 @@ Notebook environment to quickly test if your installation has succeeded
`numerblox` features the following functionality:

1. Downloading data
2. A custom data structure extending Pandas DataFrame (`NumerFrame`)
2. A custom data structure extending Pandas DataFrame
([`NumerFrame`](https://crowdcent.github.io/numerblox/numerframe.html#numerframe))
3. A suite of preprocessors for Numerai Classic and Signals (feature
selection, engineering and manipulation)
4. Model objects for easy inference.
5. A suite of postprocessors for Numerai Classic and Signals
(standardization, ensembling, neutralization and penalization)
6. Pipelines handling processing and prediction (`ModelPipeline` and
`ModelPipelineCollection`)
7. Evaluation (`NumeraiClassicEvaluator` and `NumeraiSignalsEvaluator`)
8. Authentication (`Key` and `load_key_from_json`)
9. Submitting (`NumeraiClassicSubmitter`, `NumeraiSignalsSubmitter` and
`NumerBaySubmitter`)
10. Automated staking (`NumeraiClassicStaker` and
`NumeraiSignalsStaker`)
6. Pipelines handling processing and prediction
([`ModelPipeline`](https://crowdcent.github.io/numerblox/modelpipeline.html#modelpipeline)
and
[`ModelPipelineCollection`](https://crowdcent.github.io/numerblox/modelpipeline.html#modelpipelinecollection))
7. Evaluation
([`NumeraiClassicEvaluator`](https://crowdcent.github.io/numerblox/evaluation.html#numeraiclassicevaluator)
and
[`NumeraiSignalsEvaluator`](https://crowdcent.github.io/numerblox/evaluation.html#numeraisignalsevaluator))
8. Authentication
([`Key`](https://crowdcent.github.io/numerblox/key.html#key) and
[`load_key_from_json`](https://crowdcent.github.io/numerblox/key.html#load_key_from_json))
9. Submitting
([`NumeraiClassicSubmitter`](https://crowdcent.github.io/numerblox/submission.html#numeraiclassicsubmitter),
[`NumeraiSignalsSubmitter`](https://crowdcent.github.io/numerblox/submission.html#numeraisignalssubmitter)
and
[`NumerBaySubmitter`](https://crowdcent.github.io/numerblox/submission.html#numerbaysubmitter))
10. Automated staking
([`NumeraiClassicStaker`](https://crowdcent.github.io/numerblox/staking.html#numeraiclassicstaker)
and
[`NumeraiSignalsStaker`](https://crowdcent.github.io/numerblox/staking.html#numeraisignalsstaker))

#### 2.1.2. Educational notebooks

Example notebooks can be found in the `nbs/edu_nbs` directory.

`nbs/edu_nbs` currently contains the following examples: -
`numerframe_tutorial.ipynb`: A deep dive into what `NumerFrame` has to
offer. - `pipeline_construction.ipynb`: How to use `numerblox` tools for
efficient Numerai inference. - `submitting.ipynb`: How to use Submitters
for safe and easy Numerai submissions. - `google_cloud_storage.ipynb`:
How to use Downloaders and Submitters to interact with Google Cloud
Storage (GCS). - `load_model_from_wandb.ipynb`: For [Weights &
Biases](https://wandb.ai/) users. Easily pull a model from W&B for
inference. - `numerbay_integration.ipynb`: How to use `NumerBlox` to
download and upload predictions listed on
[NumerBay](https://numerbay.ai).
`numerframe_tutorial.ipynb`: A deep dive into what
[`NumerFrame`](https://crowdcent.github.io/numerblox/numerframe.html#numerframe)
has to offer. - `pipeline_construction.ipynb`: How to use `numerblox`
tools for efficient Numerai inference. - `submitting.ipynb`: How to use
Submitters for safe and easy Numerai submissions. -
`google_cloud_storage.ipynb`: How to use Downloaders and Submitters to
interact with Google Cloud Storage (GCS). -
`load_model_from_wandb.ipynb`: For [Weights & Biases](https://wandb.ai/)
users. Easily pull a model from W&B for inference. -
`numerbay_integration.ipynb`: How to use `NumerBlox` to download and
upload predictions listed on [NumerBay](https://numerbay.ai).

Development notebooks are also in the `nbs` directory. These notebooks
are also used to generate the documentation.
138 changes: 20 additions & 118 deletions _proc/00_misc.ipynb
Original file line number Diff line number Diff line change
@@ -26,35 +26,7 @@
"metadata": {
"language": "python"
},
"outputs": [
{
"data": {
"application/javascript": [
"\n",
" setTimeout(function() {\n",
" var nbb_cell_id = 1;\n",
" var nbb_unformatted_code = \"%load_ext autoreload\\n%autoreload 2\\n%load_ext nb_black\\n%load_ext lab_black\";\n",
" var nbb_formatted_code = \"%load_ext autoreload\\n%autoreload 2\\n%load_ext nb_black\\n%load_ext lab_black\";\n",
" var nbb_cells = Jupyter.notebook.get_cells();\n",
" for (var i = 0; i < nbb_cells.length; ++i) {\n",
" if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
" if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
" nbb_cells[i].set_text(nbb_formatted_code);\n",
" }\n",
" break;\n",
" }\n",
" }\n",
" }, 500);\n",
" "
],
"text/plain": [
"<IPython.core.display.Javascript object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"outputs": [],
"source": [
"#| include: false"
]
@@ -78,6 +50,8 @@
"text/markdown": [
"---\n",
"\n",
"[source](https://github.com/crowdcent/numerblox/tree/master/blob/master/numerblox/misc.py#L7){target=\"_blank\" style=\"float:right; font-size:smaller\"}\n",
"\n",
"### AttrDict\n",
"\n",
"> AttrDict (*args, **kwargs)\n",
@@ -87,6 +61,8 @@
"text/plain": [
"---\n",
"\n",
"[source](https://github.com/crowdcent/numerblox/tree/master/blob/master/numerblox/misc.py#L7){target=\"_blank\" style=\"float:right; font-size:smaller\"}\n",
"\n",
"### AttrDict\n",
"\n",
"> AttrDict (*args, **kwargs)\n",
@@ -111,45 +87,7 @@
"metadata": {
"language": "python"
},
"outputs": [
{
"data": {
"text/plain": [
"('hello', 'world')"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"application/javascript": [
"\n",
" setTimeout(function() {\n",
" var nbb_cell_id = 4;\n",
" var nbb_unformatted_code = \"test_dict = AttrDict({\\\"test1\\\": \\\"hello\\\", \\\"test2\\\": \\\"world\\\"})\\ntest_dict.test1, test_dict['test2']\";\n",
" var nbb_formatted_code = \"test_dict = AttrDict({\\\"test1\\\": \\\"hello\\\", \\\"test2\\\": \\\"world\\\"})\\ntest_dict.test1, test_dict[\\\"test2\\\"]\";\n",
" var nbb_cells = Jupyter.notebook.get_cells();\n",
" for (var i = 0; i < nbb_cells.length; ++i) {\n",
" if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
" if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
" nbb_cells[i].set_text(nbb_formatted_code);\n",
" }\n",
" break;\n",
" }\n",
" }\n",
" }, 500);\n",
" "
],
"text/plain": [
"<IPython.core.display.Javascript object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"outputs": [],
"source": [
"test_dict = AttrDict({\"test1\": \"hello\", \"test2\": \"world\"})\n",
"test_dict.test1, test_dict['test2']"
@@ -168,65 +106,29 @@
"metadata": {
"language": "python"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Converted 00_misc.ipynb.\n",
"Converted 01_download.ipynb.\n",
"Converted 02_numerframe.ipynb.\n",
"Converted 03_preprocessing.ipynb.\n",
"Converted 04_model.ipynb.\n",
"Converted 05_postprocessing.ipynb.\n",
"Converted 06_modelpipeline.ipynb.\n",
"Converted 07_evaluation.ipynb.\n",
"Converted 08_key.ipynb.\n",
"Converted 09_submission.ipynb.\n",
"Converted 10_staking.ipynb.\n",
"Converted index.ipynb.\n"
]
},
{
"data": {
"application/javascript": [
"\n",
" setTimeout(function() {\n",
" var nbb_cell_id = 5;\n",
" var nbb_unformatted_code = \"# hide\\n# Run this cell to sync all changes with library\\nfrom nbdev import nbdev_export\\n\\nnbdev_export()\";\n",
" var nbb_formatted_code = \"# hide\\n# Run this cell to sync all changes with library\\nfrom nbdev import nbdev_export\\n\\nnbdev_export()\";\n",
" var nbb_cells = Jupyter.notebook.get_cells();\n",
" for (var i = 0; i < nbb_cells.length; ++i) {\n",
" if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
" if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
" nbb_cells[i].set_text(nbb_formatted_code);\n",
" }\n",
" break;\n",
" }\n",
" }\n",
" }, 500);\n",
" "
],
"text/plain": [
"<IPython.core.display.Javascript object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"outputs": [],
"source": [
"#| include: false\n",
"# Run this cell to sync all changes with library\n",
"from nbdev import nbdev_export\n",
"\n",
"from nbdev import nbdev_export"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"language": "python"
},
"outputs": [],
"source": [
"#| include: false\n",
"nbdev_export()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"display_name": "Python 3.8.10 64-bit",
"language": "python",
"name": "python3"
},
Loading