Testing assert/required using ThrowProxy

[marcosmartinez7]

Hi, im trying to test the requires of a Smart Contract using just solidity according to this article:

http://truffleframework.com/tutorials/testing-for-throws-in-solidity-tests

This is the contract, the throw proxy contract and the test:

/* Testing with solidity tests. */

import "truffle/Assert.sol";
import "truffle/DeployedAddresses.sol";
import "../contracts/MyContract.sol";

contract TestMyContract {

function testInitialStoredValue() {
    MyContract mycontract = new MyContract();

    uint expected = 24;

    Assert.equal(mycontract.mynumber(), expected, "First number set should be 24.");
}

function testTheThrow() {
    MyContract mycontract = new MyContract(); 

    ThrowProxy throwproxy = new ThrowProxy(address(mycontract)); 
    uint num = 7;
    MyContract(address(throwproxy)).storeNum(num);

    bool r = throwproxy.execute.gas(200000)(); 

    Assert.isFalse(r, "Should be false because is should throw!");

}

function testNoThrow() {
    MyContract mycontract = new MyContract(); 

    ThrowProxy throwproxy = new ThrowProxy(address(mycontract)); 

    MyContract(address(throwproxy)).storeNum(22);

    bool r = throwproxy.execute.gas(200000)(); 

    Assert.isTrue(r, "Should be true because is should throw!");

}

}

// Proxy contract for testing throws
contract ThrowProxy {
  address public target;
  bytes data;

  function ThrowProxy(address _target) {
    target = _target;
  }

  //prime the data using the fallback function.
  function() {
    data = msg.data;
  }

  function execute() returns (bool) {
    return target.call(data);
  }
}

If i run the tests i get this error:

If i chang the storeNum function to void from

 function storeNum(uint mynum)
        public
        returns (bool success)
    {
     require(mynum > 10);
     mynumber = mynum; 
     return true;    
    }

to

 function storeNum(uint mynum)
        public
    {
     require(mynum > 10);
     mynumber = mynum; 
     return true;    
    }

the tests works..

Any ideas?

Im using Truffle v4.1.11

[marcosmartinez7]

I have tried also with another functions that returns other types and the problem is the same, it only works for void functions

any ideas?

thanks

[cgewecke]

@marcosmartinez7 This is super interesting - the ThrowProxy article was written a while ago and I wonder if recent changes at solidity have added that constraint. Will ask around this morning and see if anyone knows the answer to this question.

Out of curiosity - do your transactional methods need return signatures? If they're used within Solidity they will automatically return true if they run without raising an exception and false otherwise.

[marcosmartinez7]

Hi @cgewecke , thanks for the answer.

There is a couple of recent (aprox 5 months ago) articles and github repos using this aproach so thats an option, i dont know if it is a constraint or the proxy aproach just doesnt work now.

I really apreciate if you can research about this!

About the methods, the code that i have posted here is just an example, but the Smart contract that i am working on is from a third party and i cannot change it, but it has the return signatures. I will ask them about that because i didnt know that they automatically return true.

Thanks!

[marcosmartinez7]

@cgewecke any news? The aproach that i have taken is to wrap all the functions that arent void functions in a test contract that expose that functions in a void way.. it is not something im proud of but works :/

[cgewecke]

@marcosmartinez7 I asked the @gnidan (another engineer here - he's recently written a solidity debugger and is intimately familiar with Solidity internals :) He was surprised by this behavior.

Below is all the information I can find about address.call in the Soldity docs (from here):

to interface with contracts that do not adhere to the ABI, the function call is provided which takes an arbitrary number of arguments of any type. These arguments are padded to 32 bytes and concatenated. One exception is the case where the first argument is encoded to exactly four bytes. In this case, it is not padded to allow the use of function signatures here.

address nameReg = 0x72ba7d8e73fe8eb666ea66babc8116a41bfb10e2;
nameReg.call("register", "MyName");
nameReg.call(bytes4(keccak256("fun(uint256)")), a);

call returns a boolean indicating whether the invoked function terminated (true) or caused an EVM exception (false). It is not possible to access the actual data returned (for this we would need to know the encoding and size in advance).

There's nothing in the release notes at Solidity that makes this clearer.

The only other thing I can think of that might be related is this thread at Solidity which addresses recent changes in the way return call values are handled.

If you find anything out, please report. Will leave this issue open in the interim and thanks for the work-around, that's very helpful.

[schemar]

Just wanted to chime in and say that I have the same problem and am also using wrapper contracts now. It appears the problem is that the fallback function does not have a return value, right?

[stale]

Thank you for raising this issue! It has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. If you would like to keep this issue open, please respond with information about the current state of this problem.

[stale]

There has been no new activity on this issue since it was marked as stale 7 days ago, so it is being automatically closed. If you'd like help with this or a different problem, please open a new issue. Thanks!

[rmlopes]

Sorry for bumping up this. I have been struggling with this issue for a couple of days now. It looks like as long as any modifier is used the ThrowProxy will fail as described. Using Solidity v0.5.

[stale]

Thanks for your response! This issue is no longer considered stale and someone from the Truffle team will try to respond as soon as they can.

[sleepingevil]

I'm having the exact same issue in a JavaScript test, using truffle 5.0.9.

function I'm testing:

function transfer (address _to, uint256 _value) public returns (bool success) {
        require(balanceOf[msg.sender] >= _value);

        balanceOf[msg.sender] -= _value;
        balanceOf[_to] += _value;

        emit Transfer(msg.sender, _to, _value);

        return true;
    }

My test:

it('returns false for an unsuccessful transfer', () => MyToken.deployed().then(async function (instance) {
        const success = await instance.transfer.call(accounts[1], 1000001, { from: accounts[0] });

        assert.equal(success, false, 'success didn\'t equal false');
    }));

Expected behavior:

The test passes.

Actual behavior:

The test fails with an error thrown: Returned error: VM Exception while processing transaction: revert

[eggplantzzz]

@sleepingevil So that is currently the correct behavior when a require fails. So if you want to test it, you could put in a try/catch and test against the error that is thrown. So it would be something like this...

try {
  myContract.methodThatReverts.call();
} catch (error) {
  let result = error === isWhatIExpectItToBe;
  assert(result);
}

I hope that helps!

[marcosmartinez7]

@eggplantzzz No, thats not the correct behaviour in Solidity. Also, there is no try-catch block in Solidity. The code that youre posting here is a js test.

[sleepingevil]

@eggplantzzz: @marcosmartinez7 is right. This is not the expected behaviour. ERC20 specification requires the transfer method of a token to:

• Throw if the sender doesn't have sufficient balance
• Return a boolean

I'm trying to test the return value in this case, not the exception thrown.

According to the throw testing documentation, you can do this in Solidity with ThrowProxy. You can also test it with contract.transfer.call(...) in javascript, which will not execute the transaction, but it will just make the method return its return value for testing purposes.

Both in Solidity and JavaScript this behavior is not as described, as the method throws a VM exception (revert).

I posted JS code here, because the expected and actual behavior are the same in both programming languages, which makes me believe that the root cause is the same. Please advise me if I should create a separate issue for the JS one (I'm just trying to avoid having duplicate issues in github).

[eggplantzzz]

@marcosmartinez7 Sorry, I was answering @sleepingevil.

From the Solidity docs it states that an exception occurs when the value in a require statement evaluates to false.
https://solidity.readthedocs.io/en/v0.5.7/control-structures.html#error-handling-assert-require-revert-and-exceptions

I guess regardless of the specs, just from looking at the Solidity code it seems like it is behaving as I would expect Solidity to behave. throw seems to have been removed as of Solidity 0.5.0. Am I missing something?

[sleepingevil]

@eggplantzzz Actually, I think I'm missing something. :)

Normally, if a function throws an exception, I'd expect it to stop executing, and don't return anything.

However, I was under the assumption that in Solidity, if a function has a boolean return value, it will still return false even if it throws.

I assumed, because that's exactly what happens in this tutorial that I was watching. Apparently that was the incorrect behavior in the version of truffle used in the video, and what we have in truffle 5 today is correct.

Thanks for the help clearing the confusion!