Errors in sequencer aren't easily visible/exported anywhere

[RJPercival]

$ go test -v github.com/google/trillian/integration
=== RUN   TestLiveLogIntegration
--- SKIP: TestLiveLogIntegration (0.00s)
        log_integration_test.go:50: Log integration test skipped as no tree ID provided
=== RUN   TestInProcessLogIntegration
E0711 11:45:35.614068   18160 log_operation_manager.go:388] failed to execute operation on logs: failed to retrieve full list of log IDs: failed to get tx for retrieving logIDs: context canceled
--- PASS: TestInProcessLogIntegration (17.52s)
=== RUN   TestInProcessLogIntegrationDuplicateLeaves
*** Test killed: ran too long (10m0s).
FAIL    github.com/google/trillian/integration  600.600s

MySQL 5.7 server installed using developer configuration.

[RJPercival]

After seeing a similar timeout occur after reducing the max number of connections to the MySQL server (in PR #722), I tried increasing the max_connections MySQL server parameter from the default of 151 to 500. This had no effect.

[RJPercival]

The cause of this was a bug in PR #734 that resulted in sequencing passes failing, but the integration test not reporting this as an error. Instead, it would just spin attempting pass after pass until the timeout was reached. The integration test should be made to abort if a terminal error occurs.

[Martin2112]

Is this still an issue? I haven't seen this fail recently.

[RJPercival]

Yes, the LogOperationManager swallows sequencing errors and so the integration tests can't detect that there's a problem - they just run until the timeout. There's a relevant TODO here: https://github.com/google/trillian/blob/master/server/log_operation_manager.go#L378

[Martin2112]

The TODO seems to no longer be there. Not sure if this is fixed.

[RJPercival]

The LogOperationManager still swallows these errors.

[Martin2112]

Fair enough.

…

On 26 January 2018 at 14:21, Rob Percival ***@***.***> wrote: The LogOperationManager still swallows these errors <https://github.com/google/trillian/blob/master/server/log_operation_manager.go#L408> . — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#733 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AMv2T7idYrfLCIbZWVSiEaGZ3sX88yBDks5tOd94gaJpZM4OUCCf> .

[RJPercival]

The TODO is now here:

trillian/server/log_operation_manager.go

Line 285 in c53ded5

// TODO(Martin2112): No mechanism for error reporting etc., this is OK for v1 but needs work

[Martin2112]

I added error metrics in #957, doesn't help tests much but you can see signer errors on production logs.

[paulmattei]

Next step: The integration test should use the new metric

[pav-kv]

Blocked on #1640.