SDK V0.7.0 Upgrade ConnectionRefusedError: [Errno 61] Connection refused

[emanuel-skai]

Hello, I'm trying to upgrade my services to use the latest version of the SDK= nillion v0.7.0

To do soI have followed these notes: https://github.com/orgs/NillionNetwork/discussions/150, Installed the new nillion-client 0.1.0 from https://pypi.org/project/nillion-client/0.1.0/ and update my env variables for the network configuration as detailed here https://docs.nillion.com/network-configuration.

I'm using a simple script to test the changes based on the example provided here https://github.com/NillionNetwork/nillion-python-starter/blob/main/quickstart_complete/client_code/secret_addition_complete.py.

However I'm adjusting the network configuration to use the testnet instead of the devnet. My client code looks as follows.

import asyncio
import os
import pytest

from nillion_client import (
    InputPartyBinding,
    Network,
    NilChainPayer,
    NilChainPrivateKey,
    OutputPartyBinding,
    Permissions,
    SecretInteger,
    VmClient,
    PrivateKey,
)
from dotenv import load_dotenv
import logging


logger = logging.getLogger(__name__)
load_dotenv()

# 1 Party running simple addition on 1 stored secret and 1 compute time secret
async def run_debug_program(int1: int, int2: int) -> dict:
    # Use the devnet configuration generated by `nillion-devnet`
    required_env_vars = [
    "NILLION_CHAIN_ID",
    "NILLION_NILCHAIN_GRPC",
    "NILLION_BOOTNODE_MULTIADDRESS"
    ]

    missing_vars = [var for var in required_env_vars if os.getenv(var) is None]

    if missing_vars:
        raise EnvironmentError(f"Missing required environment variables: {', '.join(missing_vars)}")

    # Instantiate the Network class directly using the environment variables
    try:
        network = Network(
            chain_id=os.getenv("NILLION_CHAIN_ID"),
            chain_grpc_endpoint=os.getenv("NILLION_NILCHAIN_GRPC"),
            nilvm_grpc_endpoint=os.getenv("NILLION_BOOTNODE_MULTIADDRESS")
        )

        print("Network instantiated successfully!")
        logger.info("Network instantiated successfully!")
    except Exception as e:
        raise Exception(f"Error instantiating Network class: {e}")
    

    # Create payments config and set up Nillion wallet with a private key to pay for operations
    nilchain_key: str = os.getenv("NILLION_NILCHAIN_PRIVATE_KEY")  # type: ignore
    payer = NilChainPayer(
        network,
        wallet_private_key=NilChainPrivateKey(bytes.fromhex(nilchain_key)),
        gas_limit=10000000,
    )

    # Use a random key to identify ourselves
    signing_key = PrivateKey()
    client = await VmClient.create(signing_key, network, payer)
    party_name = "Party1"
        # Set the program name and path to the compiled program
    program_name = "secret_addition"
    script_dir = os.path.dirname(os.path.abspath(__file__))
    program_mir_rel_path = (
        "../nada_programs/target/tiny_secret_addition_complete.nada.bin"
    )
    program_mir_path = os.path.abspath(
        os.path.join(script_dir, program_mir_rel_path)
    )
    logger.info(f"Program MIR Absolute Path: {program_mir_path}")



    ##### STORE PROGRAM
    print("-----STORE PROGRAM")

    # Store program
    program_mir = open(program_mir_path, "rb").read()
    program_id = await client.store_program(program_name, program_mir).invoke()

    # Print details about stored program
    print(f"Stored program_id: {program_id}")

    ##### STORE SECRETS
    print("-----STORE SECRETS")

    # Create a secret
    values = {
        "my_int1": SecretInteger(int1),
    }

    # Create a permissions object to attach to the stored secret
    permissions = Permissions.defaults_for_user(client.user_id).allow_compute(
        client.user_id, program_id
    )

    # Store a secret, passing in the receipt that shows proof of payment
    values_id = await client.store_values(
        values, ttl_days=5, permissions=permissions
    ).invoke()

    ##### COMPUTE
    print("-----COMPUTE")

    # Bind the parties in the computation to the client to set input and output parties
    input_bindings = [InputPartyBinding(party_name, client.user_id)]
    output_bindings = [OutputPartyBinding(party_name, [client.user_id])]

    # Create a computation time secret to use
    compute_time_values = {"my_int2": SecretInteger(int2)}

    # Compute, passing in the compute time values as well as the previously uploaded value.
    print(f"Invoking computation using program {program_id} and values id {values_id}")
    compute_id = await client.compute(
        program_id,
        input_bindings,
        output_bindings,
        values=compute_time_values,
        value_ids=[values_id],
    ).invoke()

    # Print compute result
    print(f"The computation was sent to the network. compute_id: {compute_id}")
    result = await client.retrieve_compute_results(compute_id).invoke()
    print(f"✅  Compute complete for compute_id {compute_id}")
    print(f"🖥️  The result is {result}")
    return result


if __name__ == "__main__":
    asyncio.run(run_debug_program())


@pytest.mark.asyncio
async def test_main():
    result = await run_debug_program()
    assert result == {"my_output": SecretInteger(510)}
    ```



However I'm getting the following error.

AttributeError: 'VmClient' object has no attribute 'leader'
WARNING:root:Invocation of cluster failed with error: [Errno 61] Connection refused, attempt: 1, sleep_time: 0.0s, timeout: 60s
WARNING:root:Invocation of cluster failed with error: [Errno 61] Connection refused, attempt: 2, sleep_time: 1s, timeout: 60s
WARNING:root:Invocation of cluster failed with error: [Errno 61] Connection refused, attempt: 3, sleep_time: 2s, timeout: 60s
WARNING:root:Invocation of cluster failed with error: [Errno 61] Connection refused, attempt: 4, sleep_time: 4s, timeout: 60s
WARNING:root:Invocation of cluster failed with error: [Errno 61] Connection refused, attempt: 5, sleep_time: 5.0s, timeout: 60s
WARNING:root:Invocation of cluster failed with error: [Errno 61] Connection refused, attempt: 6, sleep_time: 5.0s, timeout: 60s
WARNING:root:Invocation of cluster failed with error: [Errno 61] Connection refused, attempt: 7, sleep_time: 5.0s, timeout: 60s
WARNING:root:Invocation of cluster failed with error: [Errno 61] Connection refused, attempt: 8, sleep_time: 5.0s, timeout: 60s
WARNING:root:Invocation of cluster failed with error: [Errno 61] Connection refused, attempt: 9, sleep_time: 5.0s, timeout: 60s
WARNING:root:Invocation of cluster failed with error: [Errno 61] Connection refused, attempt: 10, sleep_time: 5.0s, timeout: 60s
INFO: 127.0.0.1:60149 - "POST /debug/?input_value=1&input_value_blob=t HTTP/1.1" 500 Internal Server Error
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/Users/emanuelhc/Desktop/code/nillion-mvp/mvp/services/nillion/.venv/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 411, in run_asgi
result = await app( # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/emanuelhc/Desktop/code/nillion-mvp/mvp/services/nillion/.venv/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in call
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/emanuelhc/Desktop/code/nillion-mvp/mvp/services/nillion/.venv/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in call
await super().call(scope, receive, send)
File "/Users/emanuelhc/Desktop/code/nillion-mvp/mvp/services/nillion/.venv/lib/python3.12/site-packages/starlette/applications.py", line 123, in call
await self.middleware_stack(scope, receive, send)
File "/Users/emanuelhc/Desktop/code/nillion-mvp/mvp/services/nillion/.venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in call
raise exc
File "/Users/emanuelhc/Desktop/code/nillion-mvp/mvp/services/nillion/.venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in call
await self.app(scope, receive, _send)
File "/Users/emanuelhc/Desktop/code/nillion-mvp/mvp/services/nillion/.venv/lib/python3.12/site-packages/starlette/middleware/cors.py", line 93, in call
await self.simple_response(scope, receive, send, request_headers=headers)
File "/Users/emanuelhc/Desktop/code/nillion-mvp/mvp/services/nillion/.venv/lib/python3.12/site-packages/starlette/middleware/cors.py", line 148, in simple_response
await self.app(scope, receive, send)
File "/Users/emanuelhc/Desktop/code/nillion-mvp/mvp/services/nillion/.venv/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 65, in call
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/Users/emanuelhc/Desktop/code/nillion-mvp/mvp/services/nillion/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
raise exc
File "/Users/emanuelhc/Desktop/code/nillion-mvp/mvp/services/nillion/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/Users/emanuelhc/Desktop/code/nillion-mvp/mvp/services/nillion/.venv/lib/python3.12/site-packages/starlette/routing.py", line 756, in call
await self.middleware_stack(scope, receive, send)
File "/Users/emanuelhc/Desktop/code/nillion-mvp/mvp/services/nillion/.venv/lib/python3.12/site-packages/starlette/routing.py", line 776, in app
await route.handle(scope, receive, send)
File "/Users/emanuelhc/Desktop/code/nillion-mvp/mvp/services/nillion/.venv/lib/python3.12/site-packages/starlette/routing.py", line 297, in handle
await self.app(scope, receive, send)
File "/Users/emanuelhc/Desktop/code/nillion-mvp/mvp/services/nillion/.venv/lib/python3.12/site-packages/starlette/routing.py", line 77, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/Users/emanuelhc/Desktop/code/nillion-mvp/mvp/services/nillion/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
raise exc
File "/Users/emanuelhc/Desktop/code/nillion-mvp/mvp/services/nillion/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/Users/emanuelhc/Desktop/code/nillion-mvp/mvp/services/nillion/.venv/lib/python3.12/site-packages/starlette/routing.py", line 72, in app
response = await func(request)
^^^^^^^^^^^^^^^^^^^
File "/Users/emanuelhc/Desktop/code/nillion-mvp/mvp/services/nillion/.venv/lib/python3.12/site-packages/fastapi/routing.py", line 278, in app
raw_response = await run_endpoint_function(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/emanuelhc/Desktop/code/nillion-mvp/mvp/services/nillion/.venv/lib/python3.12/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
return await dependant.call(**values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/emanuelhc/Desktop/code/nillion-mvp/mvp/services/nillion/web_app/routes/maintenance_routes.py", line 39, in debug
program_result = await run_debug_program(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/emanuelhc/Desktop/code/nillion-mvp/mvp/services/nillion/web_app/clients/debug_program_client.py", line 61, in run_debug_program
client = await VmClient.create(signing_key, network, payer)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/emanuelhc/Desktop/code/nillion-mvp/mvp/services/nillion/.venv/lib/python3.12/site-packages/nillion_client/client.py", line 317, in create
await client._async_init()
File "/Users/emanuelhc/Desktop/code/nillion-mvp/mvp/services/nillion/.venv/lib/python3.12/site-packages/nillion_client/client.py", line 246, in _async_init
self.cluster = await _invoke_with_retry(membership_service.cluster)(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/emanuelhc/Desktop/code/nillion-mvp/mvp/services/nillion/.venv/lib/python3.12/site-packages/tenacity/asyncio/init.py", line 189, in async_wrapped
return await copy(fn, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/emanuelhc/Desktop/code/nillion-mvp/mvp/services/nillion/.venv/lib/python3.12/site-packages/tenacity/asyncio/init.py", line 111, in call
do = await self.iter(retry_state=retry_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/emanuelhc/Desktop/code/nillion-mvp/mvp/services/nillion/.venv/lib/python3.12/site-packages/tenacity/asyncio/init.py", line 153, in iter
result = await action(retry_state)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/emanuelhc/Desktop/code/nillion-mvp/mvp/services/nillion/.venv/lib/python3.12/site-packages/tenacity/_utils.py", line 99, in inner
return call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/Users/emanuelhc/Desktop/code/nillion-mvp/mvp/services/nillion/.venv/lib/python3.12/site-packages/tenacity/init.py", line 418, in exc_check
raise retry_exc.reraise()
^^^^^^^^^^^^^^^^^^^
File "/Users/emanuelhc/Desktop/code/nillion-mvp/mvp/services/nillion/.venv/lib/python3.12/site-packages/tenacity/init.py", line 185, in reraise
raise self.last_attempt.result()
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/[email protected]/3.12.6/Frameworks/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/[email protected]/3.12.6/Frameworks/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/Users/emanuelhc/Desktop/code/nillion-mvp/mvp/services/nillion/.venv/lib/python3.12/site-packages/tenacity/asyncio/init.py", line 114, in call
result = await fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/emanuelhc/Desktop/code/nillion-mvp/mvp/services/nillion/.venv/lib/python3.12/site-packages/nillion_client_proto/nillion/membership/v1/init.py", line 33, in cluster
return await self._unary_unary(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/emanuelhc/Desktop/code/nillion-mvp/mvp/services/nillion/.venv/lib/python3.12/site-packages/betterproto/grpc/grpclib_client.py", line 84, in _unary_unary
await stream.send_message(request, end=True)
File "/Users/emanuelhc/Desktop/code/nillion-mvp/mvp/services/nillion/.venv/lib/python3.12/site-packages/grpclib/client.py", line 241, in send_message
await self.send_request()
File "/Users/emanuelhc/Desktop/code/nillion-mvp/mvp/services/nillion/.venv/lib/python3.12/site-packages/grpclib/client.py", line 173, in send_request
protocol = await self._channel.connect()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/emanuelhc/Desktop/code/nillion-mvp/mvp/services/nillion/.venv/lib/python3.12/site-packages/grpclib/client.py", line 748, in connect
self._protocol = await self._create_connection()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/emanuelhc/Desktop/code/nillion-mvp/mvp/services/nillion/.venv/lib/python3.12/site-packages/grpclib/client.py", line 725, in _create_connection
_, protocol = await self._loop.create_connection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "uvloop/loop.pyx", line 2039, in create_connection
File "uvloop/loop.pyx", line 2016, in uvloop.loop.Loop.create_connection
ConnectionRefusedError: [Errno 61] Connection refused

My network configuration values are:
NILLION_BOOTNODE_MULTIADDRESS=https://node-1.photon2.nillion-network.nilogy.xyz:14311
NILLION_NILCHAIN_CHAIN_ID=nillion-chain-testnet-1
NILLION_NILCHAIN_GRPC=https://testnet-nillion-grpc.lavenderfive.com

Can you please give me a hand on spotting the error here?

[jwolski2]

@emanuel-skai I repeated the steps you listed above and everything looks good on my end. The only suspicious looking thing in your output is the "500 Internal Server Error" log message. I'm wondering if maybe you caught our NilChain GRPC endpoint at a bad time, when it was temporarily unavailable. Can you try again and tell me if you get any different results?

[emanuel-skai]

Hey @jwolski2 thanks for the quick response. After trying a few more I'm still getting the same error message so I'm trying to debug deeper. I'm wonder if maybe some of the deprecated dependencies are creating an issue here. Would you mind to share your virtual environment so I can validate them.

Mine looks as follows:

annotated-types==0.7.0
anyio==4.4.0
asttokens==2.4.1
attrs==24.2.0
Authlib==1.3.1
bandit==1.7.9
base58==2.1.1
bech32==1.2.0
betterproto==2.0.0b7
black==24.8.0
boto3==1.35.0
botocore==1.35.0
certifi==2024.7.4
cffi==1.17.0
charset-normalizer==3.3.2
click==8.1.7
cosmpy==0.9.2
coverage==7.6.1
cryptography==43.0.0
distro==1.9.0
dnspython==2.6.1
dparse==0.6.4b0
ecdsa==0.19.0
email_validator==2.2.0
fastapi==0.111.1
fastapi-cli==0.0.5
filelock==3.15.4
flake8==7.1.1
fsspec==2024.6.1
googleapis-common-protos==1.63.2
grpcio==1.65.5
grpcio-tools==1.62.3
grpclib==0.4.7
h11==0.14.0
h2==4.1.0
hpack==4.0.0
httpcore==1.0.5
httptools==0.6.1
httpx==0.27.0
huggingface-hub==0.24.5
hyperframe==6.0.1
idna==3.7
iniconfig==2.0.0
isort==5.13.2
Jinja2==3.1.4
jiter==0.5.0
jmespath==1.0.1
joblib==1.4.2
jsonschema==4.23.0
jsonschema-specifications==2023.12.1
markdown-it-py==3.0.0
MarkupSafe==2.1.5
marshmallow==3.21.3
mccabe==0.7.0
mdurl==0.1.2
mpmath==1.3.0
multidict==6.1.0
mypy-extensions==1.0.0
nada_dsl==0.6.4
networkx==3.3
nillion-client==0.1.0
nillion-client-core==0.1.0
nillion-client-proto==0.1.0
nillion-python-helpers==0.3.1
numpy==2.1.0
openai==1.41.0
packaging==24.1
parsial==0.1.0
pathspec==0.12.1
pbr==6.0.0
pillow==10.4.0
platformdirs==4.2.2
pluggy==1.5.0
protobuf==4.25.4
psutil==6.0.0
pycodestyle==2.12.1
pycparser==2.22
pycryptodome==3.20.0
pydantic==2.8.2
pydantic_core==2.20.1
pyflakes==3.2.0
Pygments==2.18.0
PyMuPDF==1.24.9
PyMuPDFb==1.24.9
PyPDF2==3.0.1
pytest==8.3.2
pytest-asyncio==0.23.8
pytest-cov==5.0.0
pytest-mock==3.14.0
python-dateutil==2.9.0.post0
python-dotenv==1.0.0
python-multipart==0.0.9
PyYAML==6.0.2
referencing==0.35.1
regex==2024.7.24
requests==2.32.3
rich==13.7.1
richreports==0.2.0
rpds-py==0.20.0
ruamel.yaml==0.18.6
ruamel.yaml.clib==0.2.8
s3transfer==0.10.2
safetensors==0.4.4
safety==3.2.3
safety-schemas==0.0.3
scikit-learn==1.5.1
scipy==1.14.0
secp256k1==0.14.0
sentence-transformers==2.7.0
setuptools==72.2.0
shellingham==1.5.4
six==1.16.0
sniffio==1.3.1
sortedcontainers==2.4.0
starlette==0.37.2
stevedore==5.2.0
sympy==1.13.2
tenacity==9.0.0
threadpoolctl==3.5.0
tokenizers==0.19.1
toml==0.10.2
torch==2.4.0
tqdm==4.66.5
transformers==4.44.0
typer==0.12.4
typing_extensions==4.12.2
urllib3==2.2.2
uvicorn==0.29.0
uvloop==0.20.0
watchfiles==0.23.0
websockets==12.0

[jwolski2]

@emanuel-skai Here's the list from a fresh virtualenv. Are you also running from a virtualenv?

Package                   Version
------------------------- -----------
attrs                     24.2.0
base58                    2.1.1
bech32                    1.2.0
betterproto               2.0.0b7
certifi                   2024.8.30
cffi                      1.17.1
charset-normalizer        3.4.0
cosmpy                    0.9.2
ecdsa                     0.19.0
googleapis-common-protos  1.66.0
grpcio                    1.68.1
grpcio-tools              1.62.3
grpclib                   0.4.7
h2                        4.1.0
hpack                     4.0.0
hyperframe                6.0.1
idna                      3.10
jsonschema                4.23.0
jsonschema-specifications 2024.10.1
multidict                 6.1.0
nillion-client            0.1.0
nillion-client-core       0.1.0
nillion-client-proto      0.1.0
pip                       22.0.2
protobuf                  4.25.5
pycparser                 2.22
pycryptodome              3.21.0
python-dateutil           2.9.0.post0
python-dotenv             1.0.1
PyYAML                    6.0.2
referencing               0.35.1
requests                  2.32.3
rpds-py                   0.22.0
secp256k1                 0.14.0
setuptools                59.6.0
six                       1.16.0
tenacity                  9.0.0
typing_extensions         4.12.2
urllib3                   2.2.3