Counterparty v10.9.0 failing heathz checks

[reinamora137]

After upgrade to 10.9.0 the logs were showing:

2025-01-15T21:15:55.577+00:00 - [   ERROR] - API.waitress-9 - Health check failed: 'NoneType' object is not subscriptable
2025-01-15T21:16:05.587+00:00 - [   ERROR] - API.waitress-0 - Health check failed: 'NoneType' object is not subscriptable
2025-01-15T21:16:15.578+00:00 - [   ERROR] - API.waitress-1 - Health check failed: 'NoneType' object is not subscriptable

get_running_info v1 call

{
  "result": {
    "server_ready": true,
    "db_caught_up": true,
    "bitcoin_block_count": 879412,
    "last_block": {
      "block_index": 879412,
      "block_hash": "000000000000000000001e4027833a3db60112befb81f2814f433da53898a7eb",
      "block_time": 1736974161,
      "ledger_hash": "12536db07340716c59af3300a0e0df77a70e151a1e1d6cbf49f9baebd69a467b",
      "txlist_hash": "b942feb3cbd6a307c77f3f84f917599d00000f9dc27389dbfd210df9413afa8b",
      "messages_hash": "ccb3647ab6ca3b6ace2782219b1a9ff0922e9f4a6aaca0052f9d75e65fc49aa9",
      "previous_block_hash": "000000000000000000004b3b193a593d93acdb67e4bd9ebdb9ef05a9aa1c7fa9",
      "difficulty": 386042977,
      "transaction_count": 0
    },
    "bitcoind_caught_up": true,
    "bitcoind_blocks_behind": 0,
    "last_message_index": 18964048,
    "api_limit_rows": 1000,
    "running_testnet": false,
    "running_testnet4": false,
    "running_regtest": false,
    "version_major": 10,
    "version_minor": 9,
    "version_revision": 0,
    "uptime": 3653,
    "dockerized": true,
    "force_enabled": false
  },
  "id": 0,
  "jsonrpc": "2.0"
}

/healthz (same with check_type=heavy")

{
  "result": {
    "status": "Unhealthy"
  }
}

Sequence of Events Leading to Failure

1. Server startup begins

   • Global variable util.CURRENT_BLOCK_INDEX is initialized as None
   • API server starts and endpoints become available
   • Health check endpoint (/v2/healthz) becomes accessible

2. Background initialization starts

   • NodeStatusCheckerThread begins initialization
   • Thread needs to:
     • Connect to database
     • Get current block height from Bitcoin Core
     • Update global variables including CURRENT_BLOCK_INDEX

3. Race to First Health Check

   • External services (like Docker health check) attempt to call /v2/healthz
   • Health check tries to use util.CURRENT_BLOCK_INDEX which is still None
   • Results in "'NoneType' object is not subscriptable" error

Related Files

• counterpartycore/lib/api/util.py - Contains health check implementation
• counterpartycore/lib/api/wsgi.py - Contains NodeStatusCheckerThread

Tags

bug health-check startup initialization

[ouziel-slama]

@reinamora137 thanks for this well detailed issue!
Does this only happen at startup? /healthz works after a while or never?

[ouziel-slama]

@reinamora137 didn't manage to reproduce. Is it possible please for you to add these 2 lines:

import traceback
print(traceback.format_exc())

here https://github.com/CounterpartyXCP/counterparty-core/blob/master/counterparty-core/counterpartycore/lib/api/util.py#L71
then try /healthz and lmk what do you see in the log.
Thank you in advance!

[reinamora137]

@reinamora137 thanks for this well detailed issue! Does this only happen at startup? /healthz works after a while or never?

Thanks, will add that trace output and update here shortly. Currently the healthz never recovers