Improve rpc compatibility

[konradkonrad]

See coverage on: http://cdetr.io/eth-compat-table/

[gsalgado]

I wanted to give this one a go, but I don't get the same failures when I run the rpc tests locally. When I run them via hive[1] (with 'hive --client=pythereum:develop --test=rpc-tests') I get a ton of errors: http://sprunge.us/aOjI

I also tried running the tests directly from the rpc-tests repo, but the errors I get from that are not the same as the ones reported in the table above: http://sprunge.us/FBHG

So, now I'm wondering if the compatibility table above is out of date and, really, what needs to be fixed is what's reported in the paste above? Or am I missing something in the way I'm running those tests, and that's what's causing me to get a different set of failures?

[1] The hive master branch doesn't support pyethereum, and ethereum/hive#22 doesn't work against pyethereum/develop because the rpc-tests/genesis.json file uses tabs and python's YAML library complains about that, so I'm using https://github.com/gsalgado/hive/tree/pyethereum

[bobsummerwill]

@cdetrio will have the answers you seek, @gsalgado.

[cdetrio]

@gsalgado You ran Hive's rpc-tests, which uses old test cases from ethereum/rpc-tests.

The tests to generate the RPC compatibility table aren't (yet) in the main Hive repo, but a branch, which I just named rpc-compat: https://github.com/cdetrio/hive/tree/rpc-compat/simulators/ethereum/rpc-compat

The branch also includes a client for pyethereum, since as you mentioned, the pyeth client is not yet merged into the Hive repo. Also, rpc-compat uses test cases that currently live in (a fork of) ethereum/interfaces. The format for these test cases is described in ethereum/EIPs#217.

The command you'd use to generate the report for the compatibility table is something like:

hive --docker-noshell --sim=rpc-compat --test=NONE --client="go-ethereum:master|go-ethereum:storage-at|parity:master|cpp-ethereum:develop|pyethereum:develop" --loglevel=6 | tee ./simulator_rpc_results.json

Note that the rpc-tests folder you find in Hive is a validator, whereas rpc-compat is a simulator. There's documentation in the Hive readme on the difference between a validator and a simulator. The reason rpc-compat is a simulator and not a validator is because currently only simulators support reporting sub-results.

[bobsummerwill]

Thanks for the details, @cdetrio!

[gsalgado]

Thanks for the info, @cdetrio, that clarifies a lot of things!

I tried running hive from your branch, but it doesn't work because ethash depends on the go-ethereum/pow package, which no longer exists in go-ethereum:master -- it's been moved to go-ethereum/consensus/ethash AFAICS. I'll have a go at updating ethash to use the new package

Also, I'm wondering if it's still worth trying to get pyethapp to pass the old rpc-tests from https://github.com/ethereum/rpc-tests?

[cdetrio]

@gsalgado at a glance, it looks like the tests from ethereum/rpc-tests that are failing are expected to be failing. From the readme:

A note on tests: everything in the pending state or which requires newly created logs or transcations can't be tested, as the fixed blockchain is not mining.

For go-ethereum in Hive, if the go-ethereum master branch builds outside of Hive, then it should also be building inside Hive (sometimes Docker's cache causes problems... clearing the cache should fix it). If there are problems with Hive, open an issue on the Hive repo (would also help to document over there how you fixed it).

[gsalgado]

@cdetrio right, I won't spend more time on the ethereum/rpc-tests tests, then, but there's one failing test that might be worth fixing: the one for eth_accounts fails because pyethapp doesn't include the coinbase account there, unlike geth. Should I open an issue about it?

As for go-ethereum, the problem is that https://github.com/ethereum/ethash doesn't build against go-ethereum:master. The fix is trivial, though: ethereum/ethash#93

I've been trying to run the rpc-compat simulation from your hive branch, but so far I haven't been able to. After I've changed internal/ethash/Dockerfile to use my patched ethhash, hive failed to build the rpc-compat docker image, so I had to hack that Dockerfile as well. With the changes I made (http://sprunge.us/BKXS) it now builds all images and runs the simulation, but then simulator.py crashes with the following traceback:

("ERROR: couldn't load test file:", 'tests/tests')
    main()
  File "/validate_tests.py", line 141, in main
    run_test(test_name, sesh, client_rpc_url, mock_reporter)
  File "/validate_tests.py", line 40, in run_test
    raise Exception("ERROR: couldn't load tests file:", test_filename)
Exception: ("ERROR: couldn't load tests file:", 'tests/tests')

[gsalgado]

After messing around a bit more, I found out that was caused by the rpc-compat Dockerfile creating a nested tests/ directory under /tests, which broke validate_tests. With these changes I can finally run the rpc-compat tests via hive, so I'll start looking into what needs to be fixed in pyethapp/pyethereum

[cdetrio]

@gsalgado Sorry about that, shouldn't have left those Dockerfile lines in there. Glad you got it working!

When you work on pyeth, try a more recent branch, not the old one I was working on (some of the changes have since been merged upstream). Once change pyethapp still needs is to fix #174, then some of the workarounds in pyethapp.sh can be dropped.

Lastly, a script for copying the logs out of workspace/logs:

#!/bin/bash

for client in workspace/logs/simulations/*; do
  echo "client: $client"
  CLIENT_LOG_NAME=$(echo $client | sed -rn 's/[^\[]*\[(.*)\]/\1/p' | sed 's/\:/\-/g')
  echo "CLIENT_LOG_NAME: $CLIENT_LOG_NAME"
  for clientlog in "$client"/client-*.log; do
    echo "clientlog: $clientlog"
    cp "$clientlog" reportlogs/"$CLIENT_LOG_NAME"-client.log
  done
  cp "$client"/simulator.log reportlogs/"$CLIENT_LOG_NAME"-simulator.log
done

Those logs then go in the clientlogs folder for eth-compat-table.

btw, another pyeth/hive todo is to run the consensus tests against pyeth.

[karalabe]

Sorry for breaking ethash, we've switched over to a pure Go version. I've opened a PR against the old ethash to make it self contained and not depend on our repository any more. ethereum/ethash#95 This should solve the issues for code depending on the old implementation.

[gsalgado]

Running hive from https://github.com/gsalgado/hive/tree/rpc-compat, which uses my pyethapp branch from #196, I get all tests for the methods implemented in pyethapp passing.