Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test,ci,feat: (1) Use poetry for pkg management. (2) Improve CI. (3) Re-gen protos for v0.12.1-alpha of nibiru #53

Merged
merged 18 commits into from
Aug 12, 2022
Merged
Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
0ff7271
fix for paraams
Unique-Divine Aug 9, 2022
31713fa
Merge branch 'master' into ud/main
Unique-Divine Aug 12, 2022
5e26331
fix: make it so that make proto-gen runs even if the proto dir alread…
Unique-Divine Aug 12, 2022
35acac1
feat(network): add devnet to the Network class
Unique-Divine Aug 12, 2022
bd25bbc
test(chain_info_test): test_query_vpool_reserve_assets
Unique-Divine Aug 12, 2022
783c020
docs(README): Instructions on generating types with protobuf
Unique-Divine Aug 12, 2022
b16bdb7
proto: regenerare protos wth binary v0.12.1-alpha
Unique-Divine Aug 12, 2022
5df1400
feat: use poetry instead of pipenv b/c it's objectively better
Unique-Divine Aug 12, 2022
7926cd3
ci: Use poetry for the test workflow. Handle concurrency. Add cacheing.
Unique-Divine Aug 12, 2022
d6bb336
ci: base branch is called master, not main
Unique-Divine Aug 12, 2022
23a2d73
forgot to set repo secrets for the env vars
Unique-Divine Aug 12, 2022
6f8a0d9
fix: Remove maintenance margin ratio from perp query
matthiasmatt Aug 12, 2022
7c54950
refactor: remvoe maintenance margin ratio comment
Unique-Divine Aug 12, 2022
4039bad
Merge branch 'ud/poetry' of github.com:NibiruChain/nibiru-py into ud/…
Unique-Divine Aug 12, 2022
86d8cc6
Merge branch 'master' into ud/poetry
Unique-Divine Aug 12, 2022
db76f57
refactor(pytests): cleanup for PR comments
Unique-Divine Aug 12, 2022
2e843d4
refactor(chain_info_test): revert network name change
Unique-Divine Aug 12, 2022
93a3916
refactor(protocgen.sh): simiply using suggestion from KY
Unique-Divine Aug 12, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 55 additions & 16 deletions .github/workflows/pytests.yml
Original file line number Diff line number Diff line change
@@ -1,39 +1,78 @@
name: Run tests

on: [push]
on:
pull_request:
branches: ["master", "release/*"]
Unique-Divine marked this conversation as resolved.
Show resolved Hide resolved
push:
branches: ["master", "release/*"]

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}-${{ github.head_ref }}
cancel-in-progress: ${{ ! (github.ref == 'refs/heads/master' || startsWith(github.ref, 'refs/heads/release/')) }}

jobs:
tests:
runs-on: ubuntu-latest
env:
env:
# https://www.notion.so/nibiru/Resources-and-Repo-Configs-b31aa8074a2b419d80b0c946ed5efab0
CHAIN_ID: ${{ secrets.CHAIN_ID }}
VALIDATOR_MNEMONIC: ${{ secrets.VALIDATOR_MNEMONIC }}
HOST: ${{ secrets.HOST }}
VALIDATOR_MNEMONIC: ${{ secrets.VALIDATOR_MNEMONIC }}
GRPC_PORT: ${{ secrets.GRPC_PORT }}
LCD_PORT: ${{ secrets.LCD_PORT }}
steps:
# ----------------------------------------------
# check-out repo and set-up python
# ----------------------------------------------
- name: Check out the repo
uses: actions/checkout@v3

- name: Set up Python 3.9
uses: actions/setup-python@v2
with:
python-version: 3.9.13

- name: Run python
run: python --version && python -c "print('hello')"
# ----------------------------------------------
# Install & configure poetry
# ----------------------------------------------
- name: Install Poetry
uses: snok/install-poetry@v1
with:
virtualenvs-create: true
virtualenvs-in-project: true
installer-parallel: true
#----------------------------------------------
# load cached venv if cache exists
#----------------------------------------------
- name: Load cached venv
id: cached-poetry-dependencies
uses: actions/cache@v2
with:
path: .venv
key: venv-${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-${{ hashFiles('**/poetry.lock') }}
#----------------------------------------------
# install dependencies if cache does not exist
#----------------------------------------------
- name: Install dependencies
if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'
run: poetry install --no-interaction --no-root

- name: Install pipenv with pip
run: pip install pipenv

- id: cache-pipenv
uses: actions/cache@v1
with:
path: ~/.local/share/virtualenvs
key: ${{ runner.os }}-pipenv-${{ hashFiles('**/Pipfile.lock') }}

- name: Install dependencies
if: steps.cache-pipenv.outputs.cache-hit != 'true'
run: |
pipenv install --dev
# - name: Install pipenv with pip
Unique-Divine marked this conversation as resolved.
Show resolved Hide resolved
# run: pip install pipenv

# - id: cache-pipenv
# uses: actions/cache@v1
# with:
# path: ~/.local/share/virtualenvs
# key: ${{ runner.os }}-pipenv-${{ hashFiles('**/Pipfile.lock') }}

# - name: Install dependencies
# if: steps.cache-pipenv.outputs.cache-hit != 'true'
# run: |
# pipenv install --dev

- name: Run Python SDK tests
run: pipenv run pytest -p no:warnings
run: poetry run pytest -p no:warnings
34 changes: 0 additions & 34 deletions Pipfile

This file was deleted.

1,058 changes: 0 additions & 1,058 deletions Pipfile.lock

This file was deleted.

65 changes: 59 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,9 @@ It is intended to be used by coders, developers, technically-skilled traders, da
- [Python dependencies](#python-dependencies)
- [Running the tests](#running-the-tests)
- [Other dependencies](#other-dependencies)
- [Generating types wth protobuf](#generating-types-wth-protobuf)
- [Linting](#linting)
- [Gotchas](#gotchas)

## Documentation Website

Expand Down Expand Up @@ -90,6 +93,8 @@ Set up a `.env` file to set environment variables for the tests.
# Example configuration for the Nibiry Python SDK
HOST="..."
VALIDATOR_MNEMONIC="..."
GRPC_PORT="..."
LCD_PORT="..."
CHAIN_ID="..."
```

Expand Down Expand Up @@ -133,14 +138,35 @@ To run shell scripts and commands in the `Makefile`, you'll need to install the
sudo dnf install python3-devel autoconf automake gcc gcc-c++ libffi-devel libtool make pkgconfig
```

#### Generate proto binding & build
### Generating types wth protobuf

```sh
make proto-gen
python -m build # Run `pip install build` in case this fails
```
The objective is to run `make proto-gen`, which simply executes `scripts/protocgen.sh`.

In order to do this, you'll need to install a few packages on your system.
```sh
python -m pip install --user grpcio-tools
pip install mypy-protobuf
```

If you get a permissions error such as
```
rm: cannot remove 'proto/proto/epochs/query.proto': Permission denied
```
call `sudo chown -R [USER-NAME] proto` using the name of user directory.
You can find the value for `[USER-NAME]` quickly by running `whoami`. In other words, this should work:

```sh
sudo chown -R $(whoami) proto
```

You're done generating types once you've successfully called

```sh
make proto-gen
poetry build # equivalently, you can run `python -m build`
```

#### Linting
## Linting

Enable git hook which will perform linting before each commit:

Expand All @@ -149,3 +175,30 @@ pre-commit install
```

This will keep your code clean.


## Gotchas

The `protobuf` package must be version 3.20.x or lower. Otherwise, the following error appears at runtime.

```
nibiru/clients/__init__.py:1: in <module>
from .dex import Dex # noqa
nibiru/clients/dex.py:8: in <module>
from nibiru.proto.dex.v1 import query_pb2 as dex_type
nibiru/proto/dex/v1/query_pb2.py:16: in <module>
from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2
../../../anaconda3/envs/divine/lib/python3.9/site-packages/google/api/annotations_pb2.py:30: in <module>
from google.api import http_pb2 as google_dot_api_dot_http__pb2
../../../anaconda3/envs/divine/lib/python3.9/site-packages/google/api/http_pb2.py:48: in <module>
_descriptor.FieldDescriptor(
../../../anaconda3/envs/divine/lib/python3.9/site-packages/google/protobuf/descriptor.py:560: in __new__
_message.Message._CheckCalledFromGeneratedFile()
E TypeError: Descriptors cannot not be created directly.
E If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
E If you cannot immediately regenerate your protos, some other possible workarounds are:
E 1. Downgrade the protobuf package to 3.20.x or lower.
E 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).
E
E More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates
```
8 changes: 6 additions & 2 deletions nibiru/clients/perp.py
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,9 @@ def params(self):
Returns:
dict: The current parameters for the perpetual module
"""
proto_output: perp_type.QueryParamsResponse = self.api.Params(perp_type.QueryParamsRequest())
proto_output: perp_type.QueryParamsResponse = self.api.Params(
perp_type.QueryParamsRequest()
)
output = MessageToDict(proto_output)["params"]

sdk_dec_fields = [
Expand Down Expand Up @@ -85,7 +87,9 @@ def trader_position(self, token_pair: str, trader: str) -> dict:
trader=trader,
)

proto_output: perp_type.QueryTraderPositionResponse = self.api.QueryTraderPosition(req)
proto_output: perp_type.QueryTraderPositionResponse = (
self.api.QueryTraderPosition(req)
)
output = MessageToDict(proto_output)

position_sdk_dec_fields = [
Expand Down
43 changes: 29 additions & 14 deletions nibiru/network.py
Original file line number Diff line number Diff line change
@@ -1,21 +1,36 @@
import dataclasses
import os
from typing import Dict, Optional

@dataclasses.dataclass
class Network:
def __init__(
self,
lcd_endpoint: str,
grpc_endpoint: str,
chain_id: str,
fee_denom: str,
env: str,
):
self.lcd_endpoint = lcd_endpoint
self.grpc_endpoint = grpc_endpoint
self.chain_id = chain_id
self.fee_denom = fee_denom
self.env = env
lcd_endpoint: str
grpc_endpoint: str
chain_id: str
fee_denom: str
env: str

@classmethod
def devnet(cls) -> "Network":
raise NotImplementedError
chain_config: Dict[str, Optional[str]] = {
"HOST": os.getenv("HOST"),
"GRPC_PORT": os.getenv("GRPC_PORT"),
"LCD_PORT": os.getenv("LCD_PORT"),
"CHAIN_ID": os.getenv("CHAIN_ID"),
}
for name, env_var in chain_config.items():
if env_var is None:
raise Exception("\n".join([
f"Environment variable {name} is needed for devnet.",
f"Please set {name} in your .env file."
]))
return cls(
lcd_endpoint=f'http://{chain_config["HOST"]}.:{chain_config["LCD_PORT"]}',
grpc_endpoint=f'{chain_config["HOST"]}:{chain_config["GRPC_PORT"]}',
chain_id=chain_config["CHAIN_ID"],
fee_denom='unibi',
env="devnet",
)

@classmethod
def testnet(cls) -> "Network":
Expand Down
2 changes: 1 addition & 1 deletion nibiru/proto/__init__.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import os
import sys

sys.path.insert(0, os.path.abspath(os.path.dirname(__file__)))
sys.path.insert(0, os.path.abspath(os.path.dirname(__file__)))
35 changes: 14 additions & 21 deletions nibiru/proto/common/common_pb2.py

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions nibiru/proto/common/common_pb2_grpc.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
"""Client and server classes corresponding to protobuf-defined services."""
import grpc

Loading