sign_and_send_raw_middleware stopped working in 6.0: Object of type HexBytes is not JSON serializable

[miohtama]

• Version: 6.0.0
• Python: .5
• OS: macOS

What was wrong?

sign_and_send_raw_middleware is used with LocalAccount to send transactions with the locally imported private key, as opposed to the test EVM accounts.

However, it no longer seems to work with JSON-RPC endpoints (not sure if specifically some JSON-RPCs).

The error is in the eth_sendRawTransaction params argument that gets passed HexBytes object and Web 6.0's make_request cannot serialise this. Maybe this is why test don't catch it as EthereumTester is special.

The error:

TypeError: Could not encode to JSON: dict had unencodable value at keys: {'params': because (list had unencodable value at index: [0: because (Object of type HexBytes is not JSON serializable)])}

Traceback:

  File "/Users/moo/Library/Caches/pypoetry/virtualenvs/trade-executor-8Oz1GdY1-py3.10/lib/python3.10/site-packages/web3/contract/contract.py", line 479, in transact
    return transact_with_contract_function(
  File "/Users/moo/Library/Caches/pypoetry/virtualenvs/trade-executor-8Oz1GdY1-py3.10/lib/python3.10/site-packages/web3/contract/utils.py", line 172, in transact_with_contract_function
    txn_hash = w3.eth.send_transaction(transact_transaction)
  File "/Users/moo/Library/Caches/pypoetry/virtualenvs/trade-executor-8Oz1GdY1-py3.10/lib/python3.10/site-packages/web3/eth/eth.py", line 362, in send_transaction
    return self._send_transaction(transaction)
  File "/Users/moo/Library/Caches/pypoetry/virtualenvs/trade-executor-8Oz1GdY1-py3.10/lib/python3.10/site-packages/web3/module.py", line 68, in caller
    result = w3.manager.request_blocking(
  File "/Users/moo/Library/Caches/pypoetry/virtualenvs/trade-executor-8Oz1GdY1-py3.10/lib/python3.10/site-packages/web3/manager.py", line 231, in request_blocking
    response = self._make_request(method, params)
  File "/Users/moo/Library/Caches/pypoetry/virtualenvs/trade-executor-8Oz1GdY1-py3.10/lib/python3.10/site-packages/web3/manager.py", line 175, in _make_request
    return request_func(method, params)
  File "/Users/moo/Library/Caches/pypoetry/virtualenvs/trade-executor-8Oz1GdY1-py3.10/lib/python3.10/site-packages/web3/middleware/signing.py", line 170, in middleware
    return make_request(RPCEndpoint("eth_sendRawTransaction"), [raw_tx])
  File "/Users/moo/code/ts/trade-executor/deps/web3-ethereum-defi/eth_defi/middleware.py", line 145, in middleware
    return make_request(method, params)
  File "/Users/moo/Library/Caches/pypoetry/virtualenvs/trade-executor-8Oz1GdY1-py3.10/lib/python3.10/site-packages/web3/providers/rpc.py", line 89, in make_request
    request_data = self.encode_rpc_request(method, params)
  File "/Users/moo/Library/Caches/pypoetry/virtualenvs/trade-executor-8Oz1GdY1-py3.10/lib/python3.10/site-packages/web3/providers/base.py", line 106, in encode_rpc_request
    encoded = FriendlyJsonSerde().json_encode(rpc_dict)
  File "/Users/moo/Library/Caches/pypoetry/virtualenvs/trade-executor-8Oz1GdY1-py3.10/lib/python3.10/site-packages/web3/_utils/encoding.py", line 240, in json_encode
    raise TypeError(f"Could not encode to JSON: {exc}")
TypeError: Could not encode to JSON: dict had unencodable value at keys: {'params': because (list had unencodable value at index: [0: because (Object of type HexBytes is not JSON serializable)])}

How can it be fixed?

In sign_and_send_raw_middleware change

return make_request(RPCEndpoint("eth_sendRawTransaction"), [raw_tx])

to

return make_request(RPCEndpoint("eth_sendRawTransaction"), [raw_tx.hex()])

[miohtama]

There is a test that specifically checks for bytes so maybe I am missing something here:

https://app.circleci.com/pipelines/github/ethereum/web3.py/4424/workflows/78d44c31-7473-4bc1-b3ed-2bb9e40d1c27/jobs/299444

[kclowes]

@miohtama do you have a minimum reproducible example we can test against? I just ran using a local geth --dev provider and am not able to reproduce this on a simple value transfer using send_transaction. Here's what I did:

>>> from web3 import Web3, IPCProvider
>>> from web3.middleware import geth_poa_middleware, construct_sign_and_send_raw_middleware

>>> w3 = Web3(IPCProvider('...'))
>>> w3.middleware_onion.inject(geth_poa_middleware, layer=0)

>>> acct1 = w3.eth.coinbase

>>> w3.eth.account.enable_unaudited_hdwallet_features()
>>> seed = "SEED" 
>>> acct2 = w3.eth.account.from_mnemonic(seed)

>>> w3.middleware_onion.add(construct_sign_and_send_raw_middleware(acct2))

>>> tx2 = {
...     'from': acct2.address, 
...     'to': acct1, 
...     'value': 1000, 
...     'gas': 210000, 
...     'gasPrice': 2,
...     'nonce': w3.eth.get_transaction_count(acct1)
... }
>>> w3.eth.send_transaction(tx2)
HexBytes('0x5dec76c3e7baa3b36f31ed945355465e0e44993b6c4e5b426d4559eb4c8a1df9')

[miohtama]

Thank you. Let me do a reprod!

[hackcheek]

I'm having the same issue @miohtama how do you address it?
I guess changing the source code with .hex() as you mentioned is a good solution. But IDK if this change can break something