Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tests: List of ObjC tests #48

Closed
ricardopereira opened this issue Nov 7, 2015 · 5 comments
Closed

Tests: List of ObjC tests #48

ricardopereira opened this issue Nov 7, 2015 · 5 comments

Comments

@ricardopereira
Copy link
Contributor

Purpose: to avoid pollution on the iOS specs spreadsheet and keep record of tests to be removed or rewritten. Current state of the tests.

Legend:

  • remove ❌
  • spec don't exist but seems ok ⚠️
  • rewrite ‼️
  • fix ⭕️
  • seems ok ✅

Some of them I couldn't relate to a specific spec because most of them are testing multi aspects. Ex: Checking Channel#subscribe and Channel#unsubscribe on the same test method.

Class ARTFallbackTest

  • testAllHostsIncludedOnce: ❌ (doesn't make sense)

Class ARTHttpTest

  • testPingGoogle: ⚠️
  • testNonExistantPath: ⚠️ (I think the idea of the test is checking if the 404 code is well handled)

Class ARTLogTest

  • testLogLevelToError: ❌
  • testLogLevel: ❌
  • testLogLevelNone: ❌
  • testNoCrashWithoutCustomLogger: ❌

Already implemented in Swift.

Class ARTRealtimeAttachTest

  • testAttachOnce RTL4a: ⭕️ (incomplete, doesn't make sense, it has a flag but it never changes)
  • testSkipsFromDetachingToAttaching: ⚠️ (invoke channel.attach when it is trying to DETACHING)
  • testAttachMultipleChannels: ⭕️⚠️ (doesn't raise any XCTFail)
  • testDetach RTL5d: ⭕️ (doesn't raise any XCTFail)
  • testDetaching: ✅ (checking state order)
  • testSkipsFromAttachingToDetaching: ✅ (checking state order)
  • testDetachingIgnoresDetach RTL5a: ⭕️ (incomplete)
  • testAttachFailsOnFailedConnection RTL4b: ⭕️ (incomplete, doesn't force the failed state)
  • testAttachRestricted RTL4d: ✅
  • testAttachingChannelFails: ❌ (the testAttachRestricted already check this)
  • testAttachedChannelFails: ❌ (the testAttachRestricted already check this)
  • testChannelClosesOnClose: ⚠️ (there is no Channel#close)
  • testPresenceEnterRestricted: ‼️

Class ARTRealtimeChannelHistoryTest

  • testHistory RTL10c: ✅
  • testHistoryBothChannels RTL10d: ⭕️ (should have 2 instances of RealtimeClient)
  • testHistoryForward: ⚠️
  • testHistoryForwardPagination: ⚠️
  • testHistoryBackwardPagination: ⚠️
  • testTimeBackwards: ⚠️ (could be better)
  • testTimeForwards: ⚠️ (could be better)
  • testHistoryFromAttach: ❌ (last tests already covers what this test is covering)

Class ARTRealtimeChannelTest

  • testAttach: ❌ (already tested with ARTRealtimeAttachTest)
  • testAttachBeforeConnect: ⭕️ (doesn't avoid to attach after it is connected)
  • testAttachDetach: ⚠️
  • testAttachDetachAttach: ⚠️
  • testSubscribeUnsubscribe: ⚠️ (testing 2 specs at the same time)
  • testSuspendingDetachesChannel: ⚠️
  • testFailingFailsChannel: ⭕️ (it doesn't check the SUSPENDED state)
  • testGetChannels RTS3a: ✅
  • testGetSameChannelWithParams: ⭕️ (has forced strings)
  • testAttachFails: ⚠️ (forced failure)
  • testDetachFails: ⚠️ (forced failure)
  • testPublishTooManyInArray: ⭕️ (have some doubts about [ARTPayload getPayloadArraySizeLimit:2 modify:true])
  • testClientIdPreserved: ❌ (already implemented in Swift)

Class ARTRealtimeConnectFailTest

❌ Remove all of it because it does nothing.

Class ARTRealtimeConnectTest

❌ Remove all of it because it already exists in Swift.

Class ARTRealtimeCryptoMessageTest

  • testCBCParser RSL5b: ⚠️
  • testCBCParser256: ✅ (same parser check of testCBCParser)
  • testCaseByFileContents: ⚠️

Class ARTRealtimeCryptoTest

  • testSendEncodedMessage: ⭕️ (cipherParams is now part of ChannelOptions)
  • testSendEncodedMessageOnExistingChannel: ⭕️ (cipherParams is now part of ChannelOptions)

Class ARTRealtimeInitTest

❌ Remove all of it because it already exists in Swift.

Class ARTRealtimeMessageTest

  • multipleSendName RTL6b: ⭕️ (incomplete, add error check)
  • testSingleSendText: ⚠️ (separate subscribe and publish test)
  • testPublish_10_1000: ⚠️
  • testPublish_20_200: ⚠️
  • testMultipleText_1000_10: ⚠️
  • testEchoMessagesDefault, RTL7f: ✅
  • testEchoMessagesFalse: ✅ (variation of testEchoMessagesDefault)
  • testSubscribeAttaches: ⭕️ (assign subscribeToStateChanges callback before the subscribe, ensures the channel gets attached when invoking subscribe)
  • testMessageQueue RTL6c2 (kind of): ⭕️ (have some doubts with connectingHappened flag)
  • testConnectionIdsInMessage RTL6f (kind of): ⭕️ (have some doubts with the history call)
  • testPublishImmediate: ⭕️ (assign subscribeToStateChanges and subscribe before eventEmitter.on)
  • testPublishArray RTL6i: ⭕️ (assign subscribe before call publish)
  • testPublishWithName RTL6i1: ⭕️ (assign subscribe before call publish)
  • testSubscribeToName: ⭕️ (have some doubts: multiple tests in one spec) RTL7b

Class ARTRealtimePresenceHistoryTest

  • testPresenceHistory RSP4a: ✅
  • testForward RSP4b2: ✅
  • testSecondChannel: ⚠️
  • testWaitTextBackward: ⚠️
  • testLimitForward: ⚠️
  • testLimitBackward: ⚠️
  • testFromAttach: ⚠️ (callback hell)
  • testPresenceHistoryMultipleClients: ⚠️

Class ARTRealtimePresenceTest

  • testTwoConnections: ⚠️ (the 2 instances should receive the message published on the same channel name)
  • testEnterSimple: ⚠️
  • testEnterAttachesTheChannel RTP8d: ✅
  • testSubscribeConnects RTP6c: ✅
  • testUpdateConnects: ⚠️
  • testEnterBeforeConnect: ⚠️
  • testEnterLeaveSimple: ⚠️
  • testEnterEnter: ⚠️ (Presence#enter twice, first enter, second update)
  • testEnterUpdateSimple: ⚠️ (Presence#enter and then Presence#update)
  • testUpdateNull RTP9a: ⭕️ (incomplete, check adding data for the present member)
  • testEnterLeaveWithoutData RTP10a: ✅ (data argument omitted)
  • testUpdateNoEnter RTP9b: ✅
  • testEnterAndGet: ⭕️ (2 client instances, same channel, both entered to the channel, get presence of channel but order of clients is random)
  • testEnterNoClientId RTP8c: ⭕️ (is checking for an exception but should verify the clientId)
  • testEnterOnDetached RTP8g: ⭕️ (should check for exception)
  • testEnterOnFailed RTP8g: ⭕️ (should check for exception)
  • testLeaveAndGet: ⚠️ (check if the client has been removed when a leave occurs)
  • testLeaveNoData RTP10a: ✅ (data argument omitted)
  • testLeaveNoMessage: ⚠️
  • testLeaveWithMessage: ⚠️
  • testLeaveOnDetached RTP10d: ✅
  • testLeaveOnFailed RTP10d: ✅
  • testEnterFailsOnError: ⚠️
  • testGetFailsOnDetachedOrFailed: ⚠️ (forcing fail)
  • testEnterClient RTP14d: ✅
  • testEnterClientIdFailsOnError: ⚠️
  • testWithNoClientIdUpdateLeaveEnterAnotherClient RTP15b: ⭕️ (incomplete)
  • test250ClientsEnter: ⭕️ (change limit to 250)
  • testPresenceMap RTP2: ✅
  • testLeaveBeforeEnterThrows RTP10d: ✅
  • testSubscribeToAction: ⭕️ (used unsubscribe instead using subscribe)
  • testSyncResumes: ⚠️ (need to run it)
  • testPresenceNoSideEffects RTP15: ✅
  • testPresenceWithData: ⚠️
  • testPresenceWithDataOnLeave: ⚠️

Class ARTRealtimeRecoverTest

  • testRecoverDisconnected RTN15d: ‼️ (recovering the connection with 2 instances of RealtimeClient but using options.recover = nil. Confusing)
  • testRecoverFails: ⚠️

Class ARTRealtimeResumeTest

  • testSimple: ⚠️ (2 connections, each connected to the same channel. Disconnect and reconnect one of the connections, then use that channel to send and receive the message)
  • testSimpleDisconnected: ⚠️

Class ARTRealtimeTokenTest

❌ Remove all of it because it does nothing.

Class ARTRestAppStatsTest

❌ Remove all of it because it already exists in Swift.

Class ARTRestCapabilityTest

  • testPublishRestricted: ⭕️ (fixed capability)

Class ARTRestChannelHistoryTest

  • testTimeBackwards: ⚠️
  • testTimeForwards: ⚠️
  • testHistoryForwardPagination: ⚠️
  • testHistoryBackwardPagination: ⚠️
  • testHistoryBackwardDefault: ⚠️
  • testHistoryTwoClients: ⚠️
  • testHistoryLimit RSL2b3: ✅ (check exception for invalid 1001 items)

Class ARTRestChannelPublishTest

  • testTypesByText RSL1b: ✅ (Channels#publish, the name is implicit)
  • testPublishArray RSL1a: ⭕️ (array of messages, currently returns a single message back instead of 3)
  • testPublishTooManyInArray: ❌ (don't see any limit for the array of messages)
  • testPublishUnJsonableType: ⚠️ (makes sense for Objective-C)

Class ARTRestCryptoTest

  • testSendBinary: ⚠️ (checking NSData and NSString)
  • testSendEncodedMessage: ⚠️ (fixed string)

Class ARTRestInitTest

  • testInternetIsUp: ⚠️
  • testInitWithKey: ❌ (already implemented in Swift)
  • testInitWithNoKey: ❌
  • testInitWithKeyBad: ❌
  • testInitWithOptions: ❌
  • testInitWithOptionsEnvironment: ❌
  • testGetAuth: ❌
  • testInitWithOptionsBad: ❌
  • testRestTimeNoFallbackHost: ⭕️ (it doesn't check the fallback host)
  • testRestTime: ⭕️ (Expect local clock and server clock to be synced within 10 seconds - don't expect)
  • testDefaultAuthType: ❌

Class ARTRestPresenceTest

  • testPresence: ❌ (incomplete)
  • testHistory RSP4a: ✅ (using channel persisted:presence_fixtures)
  • testHistoryDefaultBackwards RSP4b2: ✅ (using test-app-setup.json)
  • testHistoryDirection RSP4b2: ✅ (using test-app-setup.json)
  • testPresenceLimit RSP4b3: ✅ (checks exception for up max limit)

Class ARTRestTokenTest

  • testTokenSimple: ❌ (already implemented in Swift)
  • testInitWithBadToken: ⚠️
  • testInitWithBorrowedToken: ⚠️
  • testInitWithBorrowedTokenParam: ⚠️
  • testUseTokenAuthForcesToken RSA4: ✅
  • testClientIdForcesToken RSA4: ✅
  • testAuthURLForcesToken RSA4: ✅ (doesn't ensures if the transport connects with Token auth)
  • testTTLDefaultOneHour RSA5: ✅
  • testTTL RSA5: ✅
  • testTokenExpiresGetsReissued: ⚠️
  • testSameTokenIsUsed: ⚠️
  • testToken401GetsReissued: ⚠️
  • testReissuedTokenFailReturnsError: ❌ (it modifies the keySecret, forced change)
  • testUseTokenAuthThrowsWithNoMeansToCreateToken: ❌ (already implemented in Swift)
  • testExpiredBorrowedTokenErrors: ⚠️
  • testInitWithBorrowedAuthCb: ❌ (doesn't make sense)
@ricardopereira ricardopereira changed the title Tests: List of ObjC tests to be removed or rewritten Tests: List of ObjC tests Nov 11, 2015
@ricardopereira
Copy link
Contributor Author

Running the tests

Started at 2015-11-11 12:30:20.086
Ended at 2015-11-11 13:47:39.556
Executed 165 tests, with 161 failures (0 unexpected) in 4639.331 (4639.466) seconds

It's failing on parsing test-app-setup.json.

@mattheworiordan
Copy link
Member

I t's failing on parsing test-app-setup.json
Why I know that our https://github.com/ably/ably-common/blob/master/test-resources/test-app-setup.json used to have duplicates, see ably/ably-common@8735056. Are you sure the client lib is using the latest (if at all of course)?

@mattheworiordan
Copy link
Member

In response to your issue and analysis of the tests, this is very good, thanks. I do question your allocation of ⚠️ in some cases though. For example, testHistoryForwardPagination does actually demonstrate that paging works, that the API for history requests is available, etc. I realise it may not map 1-1 with an exact spec, however it is fine to complete the spreadsheet spec with references to more than one test to demonstrate that the functionality is implemented

@ricardopereira
Copy link
Contributor Author

@mattheworiordan Yes, you right. The ⚠️ in some points is dubious. Initially, I tried to map 1-1 but the tests wasn't created that way so I decided to check the aggregation of specs. It is not easy because in some of them I needed to check a lot of specs.

@ricardopereira
Copy link
Contributor Author

Running the tests

Started at 2015-11-13 07:07:57.727
Ended at 2015-11-13 08:11:39.016
Executed 155 tests, with 58 failures (16 unexpected) in 3821.059 (3821.287) seconds

After #51.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mattheworiordan @ricardopereira