RSC15a

[ricardopereira]

No description provided.

[mattheworiordan]

This test will fail once every 120 times (I think) so I am not sure is a great test. I am not sure how to test it perfectly though, perhaps retry once making it a 1 in 14,400 chance?

[tcard]

Relevant xckd:

Meaning, it's impossible to test that random numbers are being generated. What about using a mockable random() -> Int function?

[ricardopereira]

😅 Thanks for the fix.

[tcard]

PTAL

[mattheworiordan]

He he, nice one @tcard re: xckd! 👍

[ricardopereira]

LGTM, thank you.

[ricardopereira]

I wanted to merge this but the test suite is failing.
It's failing with tests that never failed before:

...
[18:19:17]: ▸ testAttachBeforeConnect, Asynchronous wait failed: Exceeded timeout of 10 seconds, with unfulfilled expectations: "attach_before_connect".
[18:19:17]: ▸ <unknown>:0
[18:19:17]: ▸ ```
[18:19:17]: ▸ ```
[18:19:17]: ▸ testAttachFails, Asynchronous wait failed: Exceeded timeout of 10 seconds, with unfulfilled expectations: "testAttachFails".
[18:19:17]: ▸ <unknown>:0
...

The odd thing is, I tried on my machine and many tests are failing as well with Exceeded timeout of 10 seconds.

[mattheworiordan]

@ricardopereira please provide a detailed log of the protocol messages with timestamps so that we can see if this is a realtime sandbox issue or a test suite issue

[tcard]

testAttachBeforeConnect, testAttachFails don't sound like they can be affected like this, as this is REST and attach/connect is Realtime. This is most likely a sandbox issue.

[ricardopereira]

I ran the complete test suite with verbose log and I noticed a bunch of can't detach when in a failed state (ARTRealtimeChannel.m#L570). I think it's a recent change. Stack trace:

#0  0x000000010af86494 in -[ARTRealtimeChannel detach:] at Source/ARTRealtimeChannel.m:570
#1  0x000000010af7d87c in -[ARTRealtimeChannels release:callback:] at Source/ARTRealtimeChannels.m:58
#2  0x000000010af7db90 in -[ARTRealtimeChannels release:] at Source/ARTRealtimeChannels.m:69
#3  0x000000010ab1be06 in +[ARTTestUtil removeAllChannels:] at Tests/ARTTestUtil.m:265

But I think that's not what's causing failures.

[ricardopereira]

So, I found the problem. The https://sandbox-rest.ably.io:443/apps request sometimes takes more than 10 seconds and the timeout is exceeded:

Test Case '-[ARTRealtimeAttachTest testAttachFailsOnFailedConnection]' started.
2016-04-18 15:47:54.202 xctest[14706:1873194] Get app key from sandbox: 2016-04-18 14:47:54 +0000
Tests/ARTRealtimeAttachTest.m:293: error: -[ARTRealtimeAttachTest testAttachFailsOnFailedConnection] : Asynchronous wait failed: Exceeded timeout of 10 seconds, with unfulfilled expectations: "testAttachFailsOnFailedConnection".
Test Case '-[ARTRealtimeAttachTest testAttachFailsOnFailedConnection]' failed (10.028 seconds).

Most of the tests are written like this, e.g.:

__weak XCTestExpectation *expectation = [self expectationWithDescription:@"detaching_to_attaching"];
[ARTTestUtil testRealtime:^(ARTRealtime *realtime) {  //<-- https://sandbox-rest.ably.io:443/apps call
   ...
}
[self waitForExpectationsWithTimeout:[ARTTestUtil timeout] handler:nil];

where the XCTestExpectation is created before the https://sandbox-rest.ably.io:443/apps request and the timeout is easily exceeded. I will do some changes and run a full test.

[tcard]

Cool, good catch @ricardopereira. I'm merging this though, as this PR has nothing to do with that and all the RestClient tests are passing, so please issue a separate PR if you don't mind.

[ricardopereira]

@tcard Yes, I will. Just making some final tests.