Solidity test invoking function in ABI order instead of original source order

[fdouglis]

• I've asked for help in the Truffle Gitter before filing this issue.

---

Issue

The order of function calls in a Solidity test file doesn't necessarily match the original file.

Steps to Reproduce

I don't know exactly how to reproduce this. For example, I created another file with several functions, and they all execute in the order expected.

Expected Behavior

The first function in the test contract should be called, then the second, and so on.

Actual Results

In my case, the first function is called, then the third, then the fourth, and finally the second.

Environment

• Operating System: Ubuntu 16.04
• Ethereum client: ganache: installed: X-AppImage-BuildId=4bc085d0-f4f6-11a7-2bf2-7146c546a2c0 image: X-AppImage-BuildId=4bc085d0-f4f6-11a7-2bf2-7146c546a2c0
• Truffle version (truffle version): Running truffle via node doesn't offer that option. It was built from a clean download of the truffle git as of 2-23-18.
• node version (node --version): v6.11.0
• npm version (npm --version): 3.10.10

I verified that the version I was previously using does invoke the functions in the expected order.:

Solidity v0.4.18 (solc-js)

I discussed this with @gnidan in the gitter. We discussed some related changes: trufflesuite/truffle-compile#42 ... I verified that OrderABI was being called in my code. Yet the order was still fubared. Suggestions?

[cgewecke]

@fdouglis Do you have a way to reproduce? Even if you can show the 'fubared' order that might be helpful for diagnosing.

[fdouglis]

As I said in my report, I can reproduce this in my system, but I don't didn't have a recipe to tell you how to reproduce -- since when I tried to create a second file that would exhibit similar behavior, that one ran in the correct order.

But the very good news is that when you asked me essentially to try harder to come up with a simplified repeatable sample, I noticed something about the contract that broke. I had taken it as a given that I was dealing with a single contract, since in essence my test contract was the only thing I cared about. But I had used a recipe I'd found online to augment my contract with event calls through inheritance rather than simply cutting & pasting. (I'd wanted to include a separate file for each, and couldn't figure out how to do this for the test files, so I wound up pasting in place.) The one that gets the wrong order has this extra contract while without it, it runs properly.

Here is the output of truffle -- all tests pass in this simple test, but the order of the tests changes:

TestOrder
    ✓ test0 (89ms)
    ✓ test1 (254ms)
    ✓ test2 (202ms)
    ✓ test3 (1191ms)
    ✓ test4 (284ms)

  TestOrder2
    ✓ test3 (1049ms)
    ✓ test2 (128ms)
    ✓ test1 (258ms)
    ✓ test0 (145ms)
    ✓ test4 (287ms)

Here is TestOrder.sol, without the extra inheritance. The adding stuff was because my original theory was that the amount of work done by a function was affecting when it ran.

pragma solidity ^0.4.18;

import "truffle/Assert.sol";

contract TestOrder {

  // call a func that does a variable amount of work
  function addit(uint max)  internal returns (uint j) {

    uint i;
    j = 1;
    for (i = 0; i < max; i++) 
      {
	j = j + i;
      }

  }

  function test0()  public {
    uint sum = addit(2);
    // trivial assertion
    Assert.notEqual(sum, 1,  "Shouldn't be equal");
  }
  function test1()  public {
    uint sum = addit(100);
    // trivial assertion
    Assert.notEqual(sum, 1,  "Shouldn't be equal");
  }
  function test2()  public {
    uint sum = addit(10);
    // trivial assertion
    Assert.notEqual(sum, 1,  "Shouldn't be equal");
  }
  function test3()  public {
    uint sum = addit(1000);
    // trivial assertion
    Assert.notEqual(sum, 1,  "Shouldn't be equal");
  }
  function test4()  public {
    uint sum = addit(123);
    // trivial assertion
    Assert.notEqual(sum, 1,  "Shouldn't be equal");
  }
}

Finally, the one with the wrapper, TestOrder2.sol, breaks:

pragma solidity ^0.4.18;

import "truffle/Assert.sol";

contract Debuggable 
{
  // derived from:
  // https://github.com/Distense/distense-contracts/blob/a73134bfe47faf8d5d3444d14305ad77eaeab9c5/contracts/Debuggable.sol
  event LogAddress(address someAddress);
  event LogBool(bool boolean);
  event LogBytes(bytes someBytes);
  event LogBytes32(bytes32 aBytes32);
  event LogString(string aString);
  event LogUInt256(uint256 someInt);
  event LogUInt32(uint32 someInt);
  event LogUInt8(uint8 someInt);
  //------------------
  function Debuggable() public
  {
  }
}
contract TestOrder2 is Debuggable {
  // call a func that does a variable amount of work
  function addit(uint max)  internal returns (uint j) {

    uint i;
    j = 1;
    for (i = 0; i < max; i++) 
      {
	j = j + i;
      }
  }

  function test0()  public {
    uint sum = addit(2);
    // trivial assertion
    Assert.notEqual(sum, 1,  "Shouldn't be equal");
  }
  function test1()  public {
    uint sum = addit(100);
    // trivial assertion
    Assert.notEqual(sum, 1,  "Shouldn't be equal");
  }
  function test2()  public {
    uint sum = addit(10);
    // trivial assertion
    Assert.notEqual(sum, 1,  "Shouldn't be equal");
  }
  function test3()  public {
    uint sum = addit(1000);
    // trivial assertion
    Assert.notEqual(sum, 1,  "Shouldn't be equal");
  }
  function test4()  public {
    uint sum = addit(123);
    // trivial assertion
    Assert.notEqual(sum, 1,  "Shouldn't be equal");
  }
}

[cgewecke]

@fdouglis Oh cool, thank you! Will check this out . . .

[cgewecke]

@fdouglis Haven't worked out why this is happening but as a workaround it looks like if you put Debuggable in it's own file and import it into the test contract, order is restored.

The top of your file would look like:

pragma solidity ^0.4.18;

import "truffle/Assert.sol";
import "./Debuggable.sol";

contract TestOrder is Debuggable {

If you have a chance could you verify that's the case? Sorry, thanks.

[cgewecke]

@fdouglis This has a fix pending at truffle-compile 45. Not sure it will make it into the next patch release, since that's tomorrow.

Thanks again for reporting. We'll leave this issue open until it's available in a some published form.

[fdouglis]

@cgewecke How about that ... The whole reason I had it in its own contract was that I wanted to include Debuggable, but in the test directory, I muffed the syntax to include it. I could have sworn I tried the simple import "./XXX" you suggested, and failed. But it worked here, anyway, and it did run the tests in the correct order. Thanks.

[cgewecke]

@fdouglis Believe this is fixed in 4.1.3. Please re-open if not. Thanks again.