test,ci,feat: (1) Use poetry for pkg management. (2) Improve CI. (3) …

…Re-gen protos for v0.12.1-alpha of nibiru (#53)

* fix for paraams

* fix: make it so that make proto-gen runs even if the proto dir already exists

* feat(network): add devnet to the Network class

* test(chain_info_test): test_query_vpool_reserve_assets

* docs(README): Instructions on generating types with protobuf

* proto: regenerare protos wth binary v0.12.1-alpha

* feat: use poetry instead of pipenv b/c it's objectively better

* ci: Use poetry for the test workflow. Handle concurrency. Add cacheing.

commit bba204296ef2f01d137af0acfd3654920b8262ac
Author: Unique-Divine <[email protected]>
Date:   Fri Aug 12 02:48:08 2022 -0500

    add pytest as a non-dev dependency

commit bd7f20f1d45ea5ee0e84187f63aec78f7d08f9f4
Author: Unique-Divine <[email protected]>
Date:   Fri Aug 12 02:43:29 2022 -0500

    ci: pytest workflow with poetry first attemp #wip

* ci: base branch is called master, not main

* forgot to set repo secrets for the env vars

* fix: Remove maintenance margin ratio from perp query

* refactor: remvoe maintenance margin ratio comment

* refactor(pytests): cleanup for PR comments

* refactor(chain_info_test): revert network name change

* refactor(protocgen.sh): simiply using suggestion from KY

Co-authored-by: matthiasmatt <[email protected]>

.github/workflows/pytests.yml

@@ -1,39 +1,61 @@
 name: Run tests
 
-on: [push]
+on: 
+  pull_request:
+    branches: ["master"]
+  push: 
+    branches: ["master"]
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}-${{ github.head_ref }}
+  cancel-in-progress: ${{ ! (github.ref == 'refs/heads/master' || startsWith(github.ref, 'refs/heads/release/')) }}
 
 jobs:
   tests:
     runs-on: ubuntu-latest
-    env:
+    env: 
+      # https://www.notion.so/nibiru/Resources-and-Repo-Configs-b31aa8074a2b419d80b0c946ed5efab0
       CHAIN_ID: ${{ secrets.CHAIN_ID }}
-      VALIDATOR_MNEMONIC: ${{ secrets.VALIDATOR_MNEMONIC }}
       HOST: ${{ secrets.HOST }}
+      VALIDATOR_MNEMONIC: ${{ secrets.VALIDATOR_MNEMONIC }}
+      GRPC_PORT: ${{ secrets.GRPC_PORT }}
+      LCD_PORT: ${{ secrets.LCD_PORT }}
     steps:
+    # ----------------------------------------------
+    # check-out repo and set-up python
+    # ----------------------------------------------
     - name: Check out the repo
       uses: actions/checkout@v3
-
     - name: Set up Python 3.9
       uses: actions/setup-python@v2
       with:
         python-version: 3.9.13
 
     - name: Run python
       run: python --version && python -c "print('hello')"
-
-    - name: Install pipenv with pip
-      run: pip install pipenv
-
-    - id: cache-pipenv
-      uses: actions/cache@v1
+    # ----------------------------------------------
+    # Install & configure poetry
+    # ----------------------------------------------
+    - name: Install Poetry
+      uses: snok/install-poetry@v1
       with:
-        path: ~/.local/share/virtualenvs
-        key: ${{ runner.os }}-pipenv-${{ hashFiles('**/Pipfile.lock') }}
-
+        virtualenvs-create: true
+        virtualenvs-in-project: true
+        installer-parallel: true
+    #----------------------------------------------
+    #       load cached venv if cache exists
+    #----------------------------------------------
+    - name: Load cached venv
+      id: cached-poetry-dependencies
+      uses: actions/cache@v2
+      with:
+        path: .venv
+        key: venv-${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-${{ hashFiles('**/poetry.lock') }}
+    #----------------------------------------------
+    # install dependencies if cache does not exist
+    #----------------------------------------------
     - name: Install dependencies
-      if: steps.cache-pipenv.outputs.cache-hit != 'true'
-      run: |
-        pipenv install --dev
-
+      if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'
+      run: poetry install --no-interaction --no-root
     - name: Run Python SDK tests
-      run: pipenv run pytest -p no:warnings
+      run: poetry run pytest -p no:warnings

README.md

@@ -30,6 +30,9 @@ It is intended to be used by coders, developers, technically-skilled traders, da
   - [Python dependencies](#python-dependencies)
   - [Running the tests](#running-the-tests)
   - [Other dependencies](#other-dependencies)
+  - [Generating types wth protobuf](#generating-types-wth-protobuf)
+- [Linting](#linting)
+- [Gotchas](#gotchas)
 
 ## Documentation Website
 
@@ -90,6 +93,8 @@ Set up a `.env` file to set environment variables for the tests.
 # Example configuration for the Nibiry Python SDK
 HOST="..."
 VALIDATOR_MNEMONIC="..."
+GRPC_PORT="..."
+LCD_PORT="..."
 CHAIN_ID="..."
 ```
 
@@ -133,14 +138,35 @@ To run shell scripts and commands in the `Makefile`, you'll need to install the
   sudo dnf install python3-devel autoconf automake gcc gcc-c++ libffi-devel libtool make pkgconfig
   ```
 
-#### Generate proto binding & build
+### Generating types wth protobuf 
 
-  ```sh
-  make proto-gen
-  python -m build     # Run `pip install build` in case this fails
-  ```
+The objective is to run `make proto-gen`, which simply executes `scripts/protocgen.sh`.
+
+In order to do this, you'll need to install a few packages on your system.
+```sh
+python -m pip install --user grpcio-tools
+pip install mypy-protobuf
+```
+
+If you get a permissions error such as 
+```
+rm: cannot remove 'proto/proto/epochs/query.proto': Permission denied
+```
+call `sudo chown -R [USER-NAME] proto` using the name of user directory. 
+You can find the value for `[USER-NAME]` quickly by running `whoami`. In other words, this should work: 
+
+```sh
+sudo chown -R $(whoami) proto
+```
+
+You're done generating types once you've successfully called
+
+```sh
+make proto-gen
+poetry build # equivalently, you can run `python -m build`
+```
 
-#### Linting
+## Linting
 
 Enable git hook which will perform linting before each commit:
 
@@ -149,3 +175,30 @@ pre-commit install
 ```
 
 This will keep your code clean.
+
+
+## Gotchas
+
+The `protobuf` package must be version 3.20.x or lower. Otherwise, the following error appears at runtime.
+
+```
+nibiru/clients/__init__.py:1: in <module>
+    from .dex import Dex  # noqa
+nibiru/clients/dex.py:8: in <module>
+    from nibiru.proto.dex.v1 import query_pb2 as dex_type
+nibiru/proto/dex/v1/query_pb2.py:16: in <module>
+    from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2
+../../../anaconda3/envs/divine/lib/python3.9/site-packages/google/api/annotations_pb2.py:30: in <module>
+    from google.api import http_pb2 as google_dot_api_dot_http__pb2
+../../../anaconda3/envs/divine/lib/python3.9/site-packages/google/api/http_pb2.py:48: in <module>
+    _descriptor.FieldDescriptor(
+../../../anaconda3/envs/divine/lib/python3.9/site-packages/google/protobuf/descriptor.py:560: in __new__
+    _message.Message._CheckCalledFromGeneratedFile()
+E   TypeError: Descriptors cannot not be created directly.
+E   If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
+E   If you cannot immediately regenerate your protos, some other possible workarounds are:
+E    1. Downgrade the protobuf package to 3.20.x or lower.
+E    2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).
+E
+E   More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates
+```

nibiru/clients/perp.py

@@ -32,7 +32,9 @@ def params(self):
         Returns:
             dict: The current parameters for the perpetual module
         """
-        proto_output: perp_type.QueryParamsResponse = self.api.Params(perp_type.QueryParamsRequest())
+        proto_output: perp_type.QueryParamsResponse = self.api.Params(
+            perp_type.QueryParamsRequest()
+        )
         output = MessageToDict(proto_output)["params"]
 
         sdk_dec_fields = [
@@ -85,7 +87,9 @@ def trader_position(self, token_pair: str, trader: str) -> dict:
             trader=trader,
         )
 
-        proto_output: perp_type.QueryTraderPositionResponse = self.api.QueryTraderPosition(req)
+        proto_output: perp_type.QueryTraderPositionResponse = (
+            self.api.QueryTraderPosition(req)
+        )
         output = MessageToDict(proto_output)
 
         position_sdk_dec_fields = [

nibiru/network.py

@@ -1,21 +1,36 @@
+import  dataclasses
+import os
+from typing import Dict, Optional 
+
+@dataclasses.dataclass
 class Network:
-    def __init__(
-        self,
-        lcd_endpoint: str,
-        grpc_endpoint: str,
-        chain_id: str,
-        fee_denom: str,
-        env: str,
-    ):
-        self.lcd_endpoint = lcd_endpoint
-        self.grpc_endpoint = grpc_endpoint
-        self.chain_id = chain_id
-        self.fee_denom = fee_denom
-        self.env = env
+    lcd_endpoint: str
+    grpc_endpoint: str
+    chain_id: str
+    fee_denom: str
+    env: str
 
     @classmethod
     def devnet(cls) -> "Network":
-        raise NotImplementedError
+        chain_config: Dict[str, Optional[str]] = {
+            "HOST": os.getenv("HOST"),
+            "GRPC_PORT": os.getenv("GRPC_PORT"),
+            "LCD_PORT": os.getenv("LCD_PORT"),
+            "CHAIN_ID": os.getenv("CHAIN_ID"),
+        }
+        for name, env_var in chain_config.items():
+            if env_var is None: 
+                raise Exception("\n".join([
+                    f"Environment variable {name} is needed for devnet.", 
+                    f"Please set {name} in your .env file."
+                    ]))
+        return cls(
+            lcd_endpoint=f'http://{chain_config["HOST"]}.:{chain_config["LCD_PORT"]}',
+            grpc_endpoint=f'{chain_config["HOST"]}:{chain_config["GRPC_PORT"]}',
+            chain_id=chain_config["CHAIN_ID"],
+            fee_denom='unibi',
+            env="devnet",
+        )
 
     @classmethod
     def testnet(cls) -> "Network":

nibiru/proto/__init__.py

@@ -1,4 +1,4 @@
 import os
 import sys
 
-sys.path.insert(0, os.path.abspath(os.path.dirname(__file__)))
+sys.path.insert(0, os.path.abspath(os.path.dirname(__file__)))

nibiru/proto/common/common_pb2_grpc.py

@@ -1,3 +1,4 @@
 # Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
 """Client and server classes corresponding to protobuf-defined services."""
 import grpc
+