Add the ability to use local variables to store hot/coldkey passwords

[roman-opentensor]

Variable for storing password: (BT_PW_%WALLET_NAME%_%COLDKEY/HOTKEY%_{}, keyname)

Functional:

• Function for creating local env variable based on uniq keyfile path;
• Encryption/decryption password;
• Reading env vars will be applied for encrypted keyfiles only;
• Saving env vars will be applied to creating/re-criation keys with passwords;
• User will be able to see the logging messages after storing/reading password;
• Implement function for storing encrypted password for wallet->key;
• Implement function for deleting encrypted password from env ;
• add documentation;

AC:

• manual tests passed
• python tests for bittensor-wallet passed
• python tests for bittensor-cli passed
• python tests for bittensor-sdk passed

[roman-opentensor]

Brief :)

In terms of manual use, the current implementation has not changed.

The implementation contains the following updates

1. All keys have a unique identifier for creating a variable in the local environment

• wallet.coldkey_file.env_var_name, wallet.hotkey_file.env_var_name;

2. The wallet's Keyfile get 2 new methods for saving and deleting an encrypted password to/from a local environment variable

• wallet.coldkey_file.save_password_to_env(password: str) saves passed password or asking type it.
• wallet.hotkey_file.save_password_to_env(password: str)
• wallet.coldkey_file.delete_password_from_env() deletes associated password from local env variable.
• wallet.hotkey_file.delete_password_from_env()

3. Only encrypted passwords are saved in the local environment variable.

• Before saving the password, it is encrypted. After reading, it is decrypted inside the wallet.

4. All encryption/decryption methods use the password saved in the local environment variable.

• The process of checking for the presence of an encrypted password in an associative local environment variable applies to all key methods that encrypt and decrypt key data.

Property env_var_name

All keys in the wallet have a property env_var_name. This value is unique within your operating system. Below is an example of how you can get this value.

Example:
If the key is located at

/Users/example/.bittensor/wallets/hotkeys/default

then name of the local environment variable will be

BT_PW__USERS_EXAMPLE__BITTENSOR_WALLETS_HOTKEYS_DEFAULT

Example getting env_var_name:

from bittensor_wallet import Wallet

my_wallet = Wallet()
my_wallet.hotkey_file.env_var_name
Out[4]: 'BT_PW__USERS_ROMAN__BITTENSOR_WALLETS_DEFAULT_HOTKEYS_DEFAULT'
my_wallet.coldkey_file.env_var_name
Out[5]: 'BT_PW__USERS_ROMAN__BITTENSOR_WALLETS_DEFAULT_COLDKEY'

Encryption/decryption usage

The saved password is used in the following cases:

1. Unlocking encrypted cold/hot keys

• wallet.unlock_coldkey, wallet.unlock_hotkey methods;

2. Creating a wallet

• wallet.create method;

6. recreating a wallet

• wallet.recreate method;

7. regenerating a hot/cold key

• wallet.regenerate_coldkey, wallet.regenerate_hotkey methods;

8. creating a keys separately

• wallet.create_new_coldkey, wallet.create_new_hotkey methods;

9. encrypting and decrypting data for a specific key

• wallet.coldkey_file.encrypt(), wallet.hotkey_file.encrypt(), wallet.coldkey_file.decrypt(), wallet.hotkey_file.decrypt() methods.

Arguments accepted by the Wallet.create method:

"""
Checks for existing coldkeypub and hotkeys, and creates them if non-existent.

    Arguments:
        coldkey_use_password (bool): Whether to use a password for coldkey. Defaults to ``True``.
        hotkey_use_password (bool): Whether to use a password for hotkey. Defaults to ``False``.
        save_coldkey_to_env (bool): Whether to save a coldkey password to local env. Defaults to ``False``.
        save_hotkey_to_env (bool): Whether to save a hotkey password to local env. Defaults to ``False``.
        coldkey_password (Optional[str]): Coldkey password for encryption. Defaults to ``None``. If `coldkey_password` is passed, then `coldkey_use_password` is automatically ``True``.
        hotkey_password (Optional[str]): Hotkey password for encryption. Defaults to ``None``. If `hotkey_password` is passed, then `hotkey_use_password` is automatically ``True``.
        overwrite (bool): Whether to overwrite an existing keys. Defaults to ``False``.
        suppress (bool): If ``True``, suppresses the display of the keys mnemonic message. Defaults to ``False``.

    Returns:
        Wallet instance with created keys.
"""

Example of creating a wallet and saving passwords to a local environment variable:

from bittensor_wallet import Wallet
from bittensor_wallet.keyfile import  keyfile_data_is_encrypted

w = Wallet(name="new_wallet", hotkey="myhotkey")
ck_pass = "c_testing"
hk_pass = "h_testing"

w.create(save_coldkey_to_env=True, save_hotkey_to_env=True, coldkey_password=ck_pass, hotkey_password=hk_pass, overwrite=True, suppress=True)

# coldkey file data is encrypted
print(">>> coldkey file data is encrypted", keyfile_data_is_encrypted(w.coldkey_file.data))
# hotkey file data is encrypted
print(">>> hotkey file data is encrypted", keyfile_data_is_encrypted(w.hotkey_file.data))

w.unlock_coldkey()

Result of execution:

Encrypting...
The password has been saved to environment variable 'BT_PW__USERS_ROMAN__BITTENSOR_WALLETS_NEW_WALLET_COLDKEY'.
Encrypting...
The password has been saved to environment variable 'BT_PW__USERS_ROMAN__BITTENSOR_WALLETS_NEW_WALLET_HOTKEYS_MYHOTKEY'.
>>> coldkey file data is encrypted True
>>> hotkey file data is encrypted True

Decrypting...
<Keypair (address=5EF55E6DfkZec2CDxxxxxxxxxxxxxxxxxxxxxxxxxxh)>

[UselessXiaoYao]

This future is not completely implement yet at release v2.0.2 , right?

[roman-opentensor]

This future is not completely implement yet at release v2.0.2 , right?

This is implemented and included in version 2.0.2. Note that this feature (creating a key-linked environment variable) is not enabled by default. Only if you pass the required flag save_coldkey_to_env=True or save_hotkey_to_env=True respectively. Details in the PR description.)
Please describe the issue you encountered.

[UselessXiaoYao]

I used to export the password enviroment variable and the run the btcli command to do transfer. But after update bittensor to 8.0. It's not work. And I read the latest code of btwallet v2.0.2, I find the format of variable is changed, but I changed to new format, the btcli transfer also not work, just print "Decrypting...", and then exit.
Below is my shell command:
export BT_PW__HOME_UBUNTU__BITTENSOR_WALLETS_MYWALLET_COLDKEY="mypassword"; btcli w transfer --dest ANOTHER_WALLET_COLDKEY --verbose --wallet.name mywallet --amount 1.0
And the echo is :
Using the wallet path from config: /home/ubuntu/.bittensor/wallets/
Initiating transfer on network: finney
Decrypting...
(.venv) ubuntu@c2-large-x86-syd-1:~/workspace/yzp$

[roman-opentensor]

I used to export the password enviroment variable and the run the btcli command to do transfer. But after update bittensor to 8.0. It's not work. And I read the latest code of btwallet v2.0.2, I find the format of variable is changed, but I changed to new format, the btcli transfer also not work, just print "Decrypting...", and then exit. Below is my shell command: export BT_PW__HOME_UBUNTU__BITTENSOR_WALLETS_MYWALLET_COLDKEY="mypassword"; btcli w transfer --dest ANOTHER_WALLET_COLDKEY --verbose --wallet.name mywallet --amount 1.0 And the echo is : Using the wallet path from config: /home/ubuntu/.bittensor/wallets/ Initiating transfer on network: finney Decrypting... (.venv) ubuntu@c2-large-x86-syd-1:~/workspace/yzp$

Could you please provide more context regarding your environment setup? OS, python version in virtual environment, versions of bittensor, bittensor-cli, bittensor-wallet.

I ask this because I have tested this behavior on my side after your message here and it works correctly for me.

Also, keep in mind, that when you export your password via terminal, you have to export encrypted password. We don't want to store your password anywhere without encryption. For example: mypassword is /-/1$,(:!!.

You can store your password with the following code:

from bittensor_wallet import Wallet

my_wallet = Wallet()

# the same approach you can use `for hotkey_file` if this is encrypted.
boolean_result = my_wallet.coldkey_file.save_password_to_env("mypassword")

In bittensor-wallet version 2.0.2 method save_password_to_env returns True in successful case and False if something wrong. I gonna replace the return result to encrypted password. Then you can get it like this:

from bittensor_wallet import Wallet

my_wallet = Wallet()

my_encrypted_password = my_wallet.coldkey_file.save_password_to_env("mypassword")

This may seem like a bit of a complicated process. But we have a responsibility to protect user data, especially passwords.

[UselessXiaoYao]

Thanks, I tried using your python code to save encrypted password to env. And doing stake remove, but I got exception below. (I have masked the coldkey & hotkey & password & encrypted password)

(.venv) ubuntu@c2-large-x86-syd-1:~/workspace/yzp$ python3 enable_keys.py 
>>> key: BT_PW__HOME_UBUNTU__BITTENSOR_WALLETS_MYCOLDKEY_COLDKEY, value: mypassword, encrypted: /-/1$,(:!!
The password has been saved to environment variable 'BT_PW__HOME_UBUNTU__BITTENSOR_WALLETS_MYCOLDKEY_COLDKEY'.
(.venv) ubuntu@c2-large-x86-syd-1:~/workspace/yzp$ btcli st remove --all --wallet.name mycoldkey --wallet.hotkey myhotkey
Unstake all staked TAO tokens? [y/n] (n): y
Using the wallet path from config: /home/ubuntu/.bittensor/wallets/
Do you want to unstake from the following keys to lmdcd02:
    - myhotkey_address: 0.194905805 τ
 [y/n]: y
Decrypting...
Task exception was never retrieved
future: <Task finished name='Task-14' coro=<Websocket._exit_with_timer() done, defined at /home/ubuntu/workspace/yzp/.venv/lib/python3.10/site-packages/bittensor_cli/src/bittensor/async_substrate_interface.py:669> exception=AttributeError("'NoneType' object has no attribute 'close'")>
Traceback (most recent call last):
  File "/home/ubuntu/workspace/yzp/.venv/lib/python3.10/site-packages/bittensor_cli/src/bittensor/async_substrate_interface.py", line 676, in _exit_with_timer
    await self.shutdown()
  File "/home/ubuntu/workspace/yzp/.venv/lib/python3.10/site-packages/bittensor_cli/src/bittensor/async_substrate_interface.py", line 687, in shutdown
    await self.ws.close()
AttributeError: 'NoneType' object has no attribute 'close'

And my python3 & bittensor_cli & bittensor_wallet version as below:

(.venv) ubuntu@c2-large-x86-syd-1:~/workspace/yzp$ uname -a
Linux c2-large-x86-syd-1 5.15.0-118-generic #128-Ubuntu SMP Fri Jul 5 09:28:59 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
(.venv) ubuntu@c2-large-x86-syd-1:~/workspace/yzp$ python3 --version
Python 3.10.12
(.venv) ubuntu@c2-large-x86-syd-1:~/workspace/yzp$ pip show bittensor
Name: bittensor
Version: 8.2.0
Summary: bittensor
Home-page: https://github.com/opentensor/bittensor
Author: bittensor.com
Author-email: 
License: MIT
Location: /home/ubuntu/workspace/yzp/.venv/lib/python3.10/site-packages
Requires: aiohttp, bittensor-cli, bittensor-wallet, bt-decode, colorama, fastapi, msgpack-numpy-opentensor, munch, nest-asyncio, netaddr, numpy, packaging, pydantic, python-Levenshtein, python-statemachine, pyyaml, requests, retry, rich, scalecodec, setuptools, substrate-interface, uvicorn, wheel
Required-by: 
(.venv) ubuntu@c2-large-x86-syd-1:~/workspace/yzp$ pip show bittensor_cli
Name: bittensor-cli
Version: 8.2.0
Summary: Bittensor CLI
Home-page: https://github.com/opentensor/btcli
Author: bittensor.com
Author-email: 
License: MIT
Location: /home/ubuntu/workspace/yzp/.venv/lib/python3.10/site-packages
Requires: aiohttp, async-property, backoff, bittensor-wallet, bt-decode, fuzzywuzzy, GitPython, Jinja2, netaddr, numpy, pycryptodome, pytest, python-Levenshtein, PyYAML, rich, scalecodec, substrate-interface, typer, websockets, wheel
Required-by: bittensor
(.venv) ubuntu@c2-large-x86-syd-1:~/workspace/yzp$ pip show bittensor_wallet
Name: bittensor-wallet
Version: 2.0.2
Summary: 
Home-page: 
Author: 
Author-email: Roman Chkhaidze <[email protected]>
License: 
Location: /home/ubuntu/workspace/yzp/.venv/lib/python3.10/site-packages
Requires: ansible, ansible-vault, cryptography, eth-utils, munch, password-strength, py-bip39-bindings, rich, termcolor
Required-by: bittensor, bittensor-cli

[roman-opentensor]

It looks like the password was read from environment successfully. Now you have an error in btcli. This bug has recently been fixed in this PR #187. The fix will be available in the next release.

Let me clarify, your coldkey decryption password is "mypassword"? I'm clarifying, because now this error is raised when an incorrect password is provided for decrypting cold/hotkey-s.

[UselessXiaoYao]

It looks like the password was read from environment successfully. Now you have an error in btcli. This bug has recently been fixed in this PR #187. The fix will be available in the next release.

Let me clarify, your coldkey decryption password is "mypassword"? I'm clarifying, because now this error is raised when an incorrect password is provided for decrypting cold/hotkey-s.

the enable_keys.py is like below:

from bittensor_wallet import Wallet

coldkeys = ["mywallet"]

for ck in coldkeys:
    _wallet = Wallet(name=ck)
    result = _wallet.coldkey_file.save_password_to_env(ck) # assume the decryption password is same as wallet name

Is there some problems in above codes?

[roman-opentensor]

from bittensor_wallet import Wallet

coldkeys = ["mywallet"]

for ck in coldkeys:
    _wallet = Wallet(name=ck)
    result = _wallet.coldkey_file.save_password_to_env(ck) # assume the decryption password is same as wallet name

This code looks correct if the wallet name and password match.