Investigate system test failures

[callmehiphop]

We're green now, but the previous build choked on this

3) datastore querying the datastore should run a query as a stream:
Error: done() called multiple times
at Suite.<anonymous> (system-test/datastore.js:355:5)
at Suite.<anonymous> (system-test/datastore.js:236:3)
at Object.<anonymous> (system-test/datastore.js:29:1)
at Array.forEach (native)
at LOOP (fs.js:1474:14)

[stephenplusplus]

Just going to log anomaly system test failures here:

1) pubsub Topic should be listed:
      Uncaught AssertionError: 2 == 3
      + expected - actual
      -2
      +3

      at system-test/pubsub.js:75:16
      at lib/common/stream-router.js:9:4454

[callmehiphop]

good plan!

[stephenplusplus]

Addressed in #784

(running in a Docker container)

  1) Search listing documents "before all" hook:
     Uncaught TypeError: Cannot read property 'statusCode' of undefined
      at Object.parseApiResp (lib/common/util.js:166:11)
      at Object.handleResp (lib/common/util.js:139:25)
      at lib/common/util.js:572:12
      at TLSSocket.socketOnEnd (_http_client.js:300:9)
      at _stream_readable.js:908:16

This specific error could be due to the timeout being reached and the process being interrupted. We might be hitting timeouts with Search because of the inability to delete Indexes. So, every time the tests are run, we test that an index can be created. Then, our before/afters require going through the indexes for the documents we create. In the absence of an index#delete method, we will probably just have to delete whatever we create per-test to avoid making such large amounts of API requests. (Right now, I have 288 indexes)

[stephenplusplus]

(running in a Docker container)

2) storage "before all" hook:
     Uncaught TypeError: Cannot read property 'statusCode' of undefined
      at Object.parseApiResp (lib/common/util.js:166:11)
      at Object.handleResp (lib/common/util.js:139:25)
      at lib/common/util.js:572:12
      at TLSSocket.socketOnEnd (_http_client.js:300:9)
      at _stream_readable.js:908:16

[stephenplusplus]

Fact: I've just learned that if a test times out, whatever code hasn't completed yet won't be aborted. It continues processing in the background, and in our case, is possibly making many API requests.

Theory: This could cause any future test to randomly fail or produce unexpected results, as the upstream API could be feeling overwhelmed or suspicious of our requests.

[callmehiphop]

When doing a release Travis will run to jobs at once - one for the release commit and one for the tag. Both will fail and you'll get some really weird looking errors. Has this always been an issue..? We should see if it's possible to stagger the jobs.

[stephenplusplus]

I think in the future we can just push the tag commit, let Travis run, then push the tag to GH.

(Marking for inclusion in #725)

[stephenplusplus]

@callmehiphop have you noticed any test failures popping up lately? Wondering if we can call this resolved.

[callmehiphop]

A couple of timeouts here and there, but I’d say otherwise we’ve been looking pretty good.

On Aug 14, 2015, at 3:40 PM, Stephen Sawchuk [email protected] wrote:

@callmehiphop https://github.com/callmehiphop have you noticed any test failures popping up lately? Wondering if we can call this resolved.

—
Reply to this email directly or view it on GitHub #775 (comment).

[stephenplusplus]

Cool. For releases, I think we just need to push in two steps. One for master, then another for the tags. We can revisit if more weirdness pops up.

[stephenplusplus]

Just enabled 1 build at a time:

http://docs.travis-ci.com/user/customizing-the-build/#Limiting-Concurrent-Builds

[dhermes]

👍