Merge branch 'main' into fix-remove-wallet

DevReadMe.md

@@ -151,6 +151,10 @@ PORTS="5000:5000 8000:8000 1000:1000" ./scripts/run_docker start --inbound-trans
 
 Refer to [the previous section](#Running) for instructions on how to run the software.
 
+### Logging
+
+You can find more details about logging and log levels [here](Logging.md).
+
 ### Running Tests
 
 To run the ACA-Py test suite, use the following script:

Logging.md

@@ -0,0 +1,52 @@
+# Logging docs
+
+Acapy supports multiple configurations of logging.
+
+## Log level
+
+Acapy's logging is based on python's [logging lib](https://docs.python.org/3/howto/logging.html).
+Log levels `DEBUG`, `INFO` and `WARNING` are available.
+Other log levels fall back to `WARNING`.
+
+## Command Line Arguments
+
+* `--log-level` - The log level to log on std out.
+* `--log-file` - Path to a file to log to.
+
+Example:
+
+```sh
+./bin/aca-py start --log-level debug --log-file acapy.log
+```
+
+## Environment Variables
+
+The log level can be configured using the environment variable `ACAPY_LOG_LEVEL`.
+The log file can be set by `ACAPY_LOG_FILE`.
+The log config can be set by `ACAPY_LOG_CONFIG`.
+
+Example:
+
+```sh
+ACAPY_LOG_LEVEL=info ACAPY_LOG_FILE=./acapy.log ACAPY_LOG_CONFIG=./acapy_log.ini ./bin/aca-py start
+```
+
+## Acapy Config File
+
+Following parameters can be used in a configuration file like [this](demo/demo-args.yaml).
+
+```yaml
+log-level: WARNING
+debug-connections: false
+debug-presentations: false
+```
+
+Warning: debug-connections and debug-presentations must not be used in a production environment as they log also credential claims values.
+Both parameters are independent of the log level, which means:
+Also if log-level is set to WARNING, connections and presentations will be logged like in debug log level.
+
+## Log config file
+
+Find an example in [default_logging_config.ini](aries_cloudagent/config/default_logging_config.ini).
+
+You can find more detail description in the [logging documentation](https://docs.python.org/3/howto/logging.html#configuring-logging).

Multiledger.md

@@ -0,0 +1,141 @@
+# Multi-ledger in ACA-Py <!-- omit in toc -->
+
+Ability to use multiple Indy ledgers (both IndySdk and IndyVdr) for resolving a `DID` by the ACA-Py agent. For read requests, checking of multiple ledgers in parallel is done dynamically according to logic detailed in [Read Requests Ledger Selection](#read-requests). For write requests, dynamic allocation of `write_ledger` is not supported. Write ledger can be assigned using `is_write` in the [configuration](#config-properties) or using any of the `--genesis-url`, `--genesis-file`, and `--genesis-transactions` startup (ACA-Py) arguments. If no write ledger is assigned then a `ConfigError` is raised.
+
+More background information including problem statement, design (algorithm) and more can be found [here](https://docs.google.com/document/d/109C_eMsuZnTnYe2OAd02jAts1vC4axwEKIq7_4dnNVA).
+
+## Table of Contents <!-- omit in toc -->
+
+- [Usage](#usage)
+  - [Example config file:](#example-config-file)
+  - [Config properties](#config-properties)
+- [Multi-ledger Admin API](#multi-ledger-admin-api)
+- [Ledger Selection](#ledger-selection)
+  - [Read Requests](#read-requests)
+    - [For checking ledger in parallel](#for-checking-ledger-in-parallel)
+  - [Write Requests](#write-requests)
+- [Impact on other ACA-Py function](#impact-on-other-aca-py-function)
+
+## Usage
+
+Multi-ledger is disabled by default. You can enable support for multiple ledgers using the `--genesis-transactions-list` startup parameter. This parameter accepts a string which is the path to the `YAML` configuration file. For example:
+
+`--genesis-transactions-list ./aries_cloudagent/config/multi_ledger_config.yml`
+
+If `--genesis-transactions-list` is specified, then `--genesis-url, --genesis-file, --genesis-transactions` should not be specified.
+
+### Example config file:
+```
+- id: localVON
+  is_production: false
+  genesis_url: 'http://host.docker.internal:9000/genesis'
+- id: bcorvinTest
+  is_production: true
+  is_write: true
+  genesis_url: 'http://test.bcovrin.vonx.io/genesis'
+```
+
+### Config properties
+For each ledger, the required properties are as following:
+
+- `id`\*: The id (or name) of the ledger, can also be used as the pool name if none provided
+- `is_production`\*: Whether the ledger is a production ledger. This is used by the pool selector algorithm to know which ledger to use for certain interactions (i.e. prefer production ledgers over non-production ledgers)
+
+For connecting to ledger, one of the following needs to be specified:
+
+- `genesis_file`: The path to the genesis file to use for connecting to an Indy ledger.
+- `genesis_transactions`: String of genesis transactions to use for connecting to an Indy ledger.
+- `genesis_url`: The url from which to download the genesis transactions to use for connecting to an Indy ledger.
+
+Optional properties:
+- `pool_name`: name of the indy pool to be opened
+- `keepalive`: how many seconds to keep the ledger open
+- `socks_proxy`
+- `is_write`: Whether the ledger is the write ledger. Only one ledger can be assigned, otherwise a `ConfigError` is raised.
+
+
+## Multi-ledger Admin API
+
+Multi-ledger related actions are grouped under the `ledger` topic in the SwaggerUI or under `/ledger/multiple` path.
+
+- `/ledger/multiple/config`:
+Returns the multiple ledger configuration currently in use
+- `/ledger/multiple/get-write-ledger`:
+Returns the current active/set `write_ledger's` `ledger_id`
+
+## Ledger Selection
+
+### Read Requests
+
+The following process is executed for these functions in ACA-Py:
+1. `get_schema`
+2. `get_credential_definition`
+3. `get_revoc_reg_def`
+4. `get_revoc_reg_entry`
+5. `get_key_for_did`
+6. `get_all_endpoints_for_did`
+7. `get_endpoint_for_did`
+8. `get_nym_role`
+9. `get_revoc_reg_delta`
+
+If multiple ledgers are configured then `IndyLedgerRequestsExecutor` service extracts `DID` from the record identifier and executes the [check](#for-checking-ledger-in-parallel) below, else it returns the `BaseLedger` instance.
+
+#### For checking ledger in parallel
+
+- `lookup_did_in_configured_ledgers` function
+  - If the calling function (above) is in [1-4], then check the `DID` in `cache` for a corresponding applicable `ledger_id`. If found, return the ledger info, else continue.
+  - Otherwise, launch parallel `_get_ledger_by_did` tasks for each of the configured ledgers.
+  - As these tasks get finished, construct `applicable_prod_ledgers` and `applicable_non_prod_ledgers` dictionaries, each with `self_certified` and `non_self_certified` inner dict which are sorted by the original order or index. 
+  - Order/preference for selection: `self_certified` > `production` > `non_production`
+    - Checks `production` ledger where the `DID` is `self_certified`
+    - Checks `non_production` ledger where the `DID` is `self_certified`
+    - Checks `production` ledger where the `DID` is not `self_certified`
+    - Checks `non_production` ledger where the `DID` is not `self_certified`
+  - Return an applicable ledger if found, else raise an exception.
+- `_get_ledger_by_did` function
+  - Build and submit `GET_NYM`
+  - Wait for a response for 10 seconds, if timed out return None
+  - Parse response
+  - Validate state proof
+  - Check if `DID` is self certified
+  - Returns ledger info to `lookup_did_in_configured_ledgers`
+
+### Write Requests
+
+On startup, the first configured applicable ledger is assigned as the `write_ledger` [`BaseLedger`], the selection is dependant on the order (top-down) and whether it is `production` or `non_production`. For instance, considering this [example configuration](#example-config-file), ledger `bcorvinTest` will be set as `write_ledger` as it is the topmost `production` ledger. If no `production` ledgers are included in configuration then the topmost `non_production` ledger is selected.
+
+## Impact on other ACA-Py function
+
+There should be no impact/change in functionality to any ACA-Py protocols.
+
+`IndySdkLedger` was refactored by replacing `wallet: IndySdkWallet` instance variable with `profile: Profile` and accordingly `.aries_cloudagent/indy/credex/verifier`, `.aries_cloudagent/indy/models/pres_preview`, `.aries_cloudagent/indy/sdk/profile.py`, `.aries_cloudagent/indy/sdk/verifier`, `./aries_cloudagent/indy/verifier` were also updated.
+
+Added `build_and_return_get_nym_request` and `submit_get_nym_request` helper functions to `IndySdkLedger` and `IndyVdrLedger`.
+
+Best practice/feedback emerging from `Askar session deadlock` issue and `endorser refactoring` PR was also addressed here by not leaving sessions open unnecessarily and changing `context.session` to `context.profile.session`, etc. 
+
+These changes are made here:
+- `./aries_cloudagent/ledger/routes.py`
+- `./aries_cloudagent/messaging/credential_definitions/routes.py`
+- `./aries_cloudagent/messaging/schemas/routes.py`
+- `./aries_cloudagent/protocols/actionmenu/v1_0/routes.py`
+- `./aries_cloudagent/protocols/actionmenu/v1_0/util.py`
+- `./aries_cloudagent/protocols/basicmessage/v1_0/routes.py`
+- `./aries_cloudagent/protocols/coordinate_mediation/v1_0/handlers/keylist_handler.py`
+- `./aries_cloudagent/protocols/coordinate_mediation/v1_0/routes.py`
+- `./aries_cloudagent/protocols/endorse_transaction/v1_0/routes.py`
+- `./aries_cloudagent/protocols/introduction/v0_1/handlers/invitation_handler.py`
+- `./aries_cloudagent/protocols/introduction/v0_1/routes.py`
+- `./aries_cloudagent/protocols/issue_credential/v1_0/handlers/credential_issue_handler.py`
+- `./aries_cloudagent/protocols/issue_credential/v1_0/handlers/credential_offer_handler.py`
+- `./aries_cloudagent/protocols/issue_credential/v1_0/handlers/credential_proposal_handler.py`
+- `./aries_cloudagent/protocols/issue_credential/v1_0/handlers/credential_request_handler.py`
+- `./aries_cloudagent/protocols/issue_credential/v1_0/routes.py`
+- `./aries_cloudagent/protocols/issue_credential/v2_0/routes.py`
+- `./aries_cloudagent/protocols/present_proof/v1_0/handlers/presentation_handler.py`
+- `./aries_cloudagent/protocols/present_proof/v1_0/handlers/presentation_proposal_handler.py`
+- `./aries_cloudagent/protocols/present_proof/v1_0/handlers/presentation_request_handler.py`
+- `./aries_cloudagent/protocols/present_proof/v1_0/routes.py`
+- `./aries_cloudagent/protocols/trustping/v1_0/routes.py`
+- `./aries_cloudagent/resolver/routes.py`
+- `./aries_cloudagent/revocation/routes.py`

aries_cloudagent/askar/profile.py

@@ -64,23 +64,23 @@ def init_ledger_pool(self):
         if self.settings.get("ledger.disabled"):
             LOGGER.info("Ledger support is disabled")
             return
-
-        pool_name = self.settings.get("ledger.pool_name", "default")
-        keepalive = int(self.settings.get("ledger.keepalive", 5))
-        read_only = bool(self.settings.get("ledger.read_only", False))
-        socks_proxy = self.settings.get("ledger.socks_proxy")
-        if read_only:
-            LOGGER.error("Note: setting ledger to read-only mode")
-        genesis_transactions = self.settings.get("ledger.genesis_transactions")
-        cache = self.context.injector.inject_or(BaseCache)
-        self.ledger_pool = IndyVdrLedgerPool(
-            pool_name,
-            keepalive=keepalive,
-            cache=cache,
-            genesis_transactions=genesis_transactions,
-            read_only=read_only,
-            socks_proxy=socks_proxy,
-        )
+        if self.settings.get("ledger.genesis_transactions"):
+            pool_name = self.settings.get("ledger.pool_name", "default")
+            keepalive = int(self.settings.get("ledger.keepalive", 5))
+            read_only = bool(self.settings.get("ledger.read_only", False))
+            socks_proxy = self.settings.get("ledger.socks_proxy")
+            if read_only:
+                LOGGER.error("Note: setting ledger to read-only mode")
+            genesis_transactions = self.settings.get("ledger.genesis_transactions")
+            cache = self.context.injector.inject_or(BaseCache)
+            self.ledger_pool = IndyVdrLedgerPool(
+                pool_name,
+                keepalive=keepalive,
+                cache=cache,
+                genesis_transactions=genesis_transactions,
+                read_only=read_only,
+                socks_proxy=socks_proxy,
+            )
 
     def bind_providers(self):
         """Initialize the profile-level instance providers."""
@@ -118,6 +118,7 @@ def bind_providers(self):
             injector.bind_provider(
                 BaseLedger, ClassProvider(IndyVdrLedger, self.ledger_pool, ref(self))
             )
+        if self.ledger_pool or self.settings.get("ledger.ledger_config_list"):
             injector.bind_provider(
                 IndyVerifier,
                 ClassProvider(

aries_cloudagent/askar/tests/test_profile.py

@@ -26,6 +26,7 @@ async def test_remove_success(self, AskarOpenStore):
         context.settings = {
             "multitenant.wallet_type": "askar-profile",
             "wallet.askar_profile": profile_id,
+            "ledger.genesis_transactions": mock.MagicMock(),
         }
         askar_profile = AskarProfile(openStore, context)
         remove_profile_stub = asyncio.Future()

aries_cloudagent/commands/provision.py

@@ -7,7 +7,11 @@
 from ..config import argparse as arg
 from ..config.default_context import DefaultContextBuilder
 from ..config.base import BaseError
-from ..config.ledger import get_genesis_transactions, ledger_config
+from ..config.ledger import (
+    get_genesis_transactions,
+    ledger_config,
+    load_multiple_genesis_transactions_from_config,
+)
 from ..config.util import common_config
 from ..config.wallet import wallet_config
 from ..protocols.coordinate_mediation.mediation_invite_store import (
@@ -36,7 +40,14 @@ async def provision(settings: dict):
     context = await context_builder.build_context()
 
     try:
-        await get_genesis_transactions(context.settings)
+        if context.settings.get("ledger.ledger_config_list"):
+            await load_multiple_genesis_transactions_from_config(context.settings)
+        if (
+            context.settings.get("ledger.genesis_transactions")
+            or context.settings.get("ledger.genesis_file")
+            or context.settings.get("ledger.genesis_url")
+        ):
+            await get_genesis_transactions(context.settings)
 
         root_profile, public_did = await wallet_config(context, provision=True)
 

aries_cloudagent/config/argparse.py

@@ -748,24 +748,49 @@ def add_arguments(self, parser: ArgumentParser):
                 "connect to the public (outside of corporate network) ledger pool"
             ),
         )
+        parser.add_argument(
+            "--genesis-transactions-list",
+            type=str,
+            required=False,
+            dest="genesis_transactions_list",
+            metavar="<genesis-transactions-list>",
+            env_var="ACAPY_GENESIS_TRANSACTIONS_LIST",
+            help=(
+                "Load YAML configuration for connecting to multiple"
+                " HyperLedger Indy ledgers."
+            ),
+        )
 
     def get_settings(self, args: Namespace) -> dict:
         """Extract ledger settings."""
         settings = {}
         if args.no_ledger:
             settings["ledger.disabled"] = True
         else:
+            configured = False
             if args.genesis_url:
                 settings["ledger.genesis_url"] = args.genesis_url
+                configured = True
             elif args.genesis_file:
                 settings["ledger.genesis_file"] = args.genesis_file
+                configured = True
             elif args.genesis_transactions:
                 settings["ledger.genesis_transactions"] = args.genesis_transactions
-            else:
+                configured = True
+            if args.genesis_transactions_list:
+                with open(args.genesis_transactions_list, "r") as stream:
+                    txn_config_list = yaml.safe_load(stream)
+                    ledger_config_list = []
+                    for txn_config in txn_config_list:
+                        ledger_config_list.append(txn_config)
+                    settings["ledger.ledger_config_list"] = ledger_config_list
+                    configured = True
+            if not configured:
                 raise ArgsParseError(
-                    "One of --genesis-url --genesis-file or --genesis-transactions "
-                    "must be specified (unless --no-ledger is specified to "
-                    "explicitly configure aca-py to run with no ledger)."
+                    "One of --genesis-url --genesis-file, --genesis-transactions "
+                    "or --genesis-transactions-list must be specified (unless "
+                    "--no-ledger is specified to explicitly configure aca-py to"
+                    " run with no ledger)."
                 )
             if args.ledger_pool_name:
                 settings["ledger.pool_name"] = args.ledger_pool_name