Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Manual testing on RTN14 and RTN15 #27

Merged
merged 12 commits into from
Sep 22, 2020
177 changes: 177 additions & 0 deletions ManualTesting.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,177 @@
This document aims to list our all the inputs from how some features are tested manually
and feasible tests can be converted to automated tests.

This document currently targets **[RTN14](https://docs.ably.io/client-lib-development-guide/features/#RTN14)**
and **[RTN15](https://docs.ably.io/client-lib-development-guide/features/#RTN15)**
sections of the spec, partially

## [RTN14](https://docs.ably.io/client-lib-development-guide/features/#RTN14)

### RTN14a ✅

**Actions performed**
1. Create realtime using an Invalid API Key
2. attach `connectionStateChange` listeners and `channelStateChange` listeners
3. connect to realtime instance using `realtime.connect()`

Logs
```js
2020-07-31 17:31:29.424647 : ConnectionStateChange event: ConnectionEvent.connecting
Reason: null
2020-07-31 17:31:30.115235 : ConnectionStateChange event: ConnectionEvent.failed
Reason: ErrorInfo message=Invalid key in request: asdf_d231:fawefsd-afwesd. (See https://help.ably.io/error/40005 for help.) code=40005 statusCode=400 href=https://help.ably.io/error/40005
```

Result: As expected

### RTN14b, RTN14g ⛔
Ignoring as these spec items for now as they revolve around `ProtocolMessage`

### RTN14c, RTN14d, RTN14e, RTN14f ✅

**Turning wifi off** - reconnect triggers every 15 seconds

loops between states: `connecting` and `disconnected` every 15s for 60s (i.e., until the default `connectionStateTtl`)
there-further, state changes between `connecting` and `suspended` continuously.

logs
```js
2020-07-31 16:44:48.912345 : ConnectionStateChange event: ConnectionEvent.connecting
Reason: null
2020-07-31 16:44:48.965277 : ConnectionStateChange event: ConnectionEvent.disconnected
Reason: ErrorInfo message=sandbox-realtime.ably.io code=80000 statusCode=503 href=https://help.ably.io/error/80000
2020-07-31 16:45:03.950481 : ConnectionStateChange event: ConnectionEvent.connecting
Reason: null
2020-07-31 16:45:04.014063 : ConnectionStateChange event: ConnectionEvent.disconnected
Reason: ErrorInfo message=sandbox-realtime.ably.io code=80000 statusCode=503 href=https://help.ably.io/error/80000
2020-07-31 16:45:18.988102 : ConnectionStateChange event: ConnectionEvent.connecting
Reason: null
2020-07-31 16:45:19.062352 : ConnectionStateChange event: ConnectionEvent.disconnected
Reason: ErrorInfo message=sandbox-realtime.ably.io code=80000 statusCode=503 href=https://help.ably.io/error/80000
2020-07-31 16:45:34.020740 : ConnectionStateChange event: ConnectionEvent.connecting
Reason: null
2020-07-31 16:45:34.109471 : ConnectionStateChange event: ConnectionEvent.disconnected
Reason: ErrorInfo message=sandbox-realtime.ably.io code=80000 statusCode=503 href=https://help.ably.io/error/80000
2020-07-31 16:45:49.056232 : ConnectionStateChange event: ConnectionEvent.connecting
Reason: null
2020-07-31 16:45:49.127278 : ConnectionStateChange event: ConnectionEvent.suspended
Reason: ErrorInfo message=sandbox-realtime.ably.io code=80000 statusCode=503 href=https://help.ably.io/error/80000
2020-07-31 16:46:49.101562 : ConnectionStateChange event: ConnectionEvent.connecting
Reason: null
2020-07-31 16:46:49.178212 : ConnectionStateChange event: ConnectionEvent.suspended
Reason: ErrorInfo message=sandbox-realtime.ably.io code=80000 statusCode=503 href=https://help.ably.io/error/80000
2020-07-31 16:47:49.139349 : ConnectionStateChange event: ConnectionEvent.connecting
Reason: null
2020-07-31 16:47:49.213962 : ConnectionStateChange event: ConnectionEvent.suspended
Reason: ErrorInfo message=sandbox-realtime.ably.io code=80000 statusCode=503 href=https://help.ably.io/error/80000
```
result: As expected

## [RTN15](https://docs.ably.io/client-lib-development-guide/features/#RTN15)

### RTN15h, RTN15h1, RTN15h2 ✅

#### Temporary key ✅

Created a temporary key on web console and deleted that after connection is established.
Connection disconnected, and eventually status changed to failed after 1 retry.

```js
2020-07-31 18:25:05.070597 : ConnectionStateChange event: ConnectionEvent.connecting
Reason: null
2020-07-31 18:25:05.296538 : ConnectionStateChange event: ConnectionEvent.connected
Reason: null
2020-07-31 18:25:12.123308 : ConnectionStateChange event: ConnectionEvent.disconnected
Reason: ErrorInfo message=Key/token status changed (revoke). (See https://help.ably.io/error/40131 for help.) code=40131 statusCode=401 href=https://help.ably.io/error/40131
2020-07-31 18:25:12.126130 : ConnectionStateChange event: ConnectionEvent.connecting
Reason: null
2020-07-31 18:25:12.408257 : ConnectionStateChange event: ConnectionEvent.failed
Reason: ErrorInfo message=Key revoked. (See https://help.ably.io/error/40131 for help.) code=40131 statusCode=401 href=https://help.ably.io/error/40131
```
result: As expected

#### Temporary app ✅

Created a temporary app on web console and deleted that after connection is established.
Connection disconnected, and eventually status changed to failed after 1 retry.

```js
2020-07-31 18:14:26.818151 : ConnectionStateChange event: ConnectionEvent.connecting
Reason: null
2020-07-31 18:14:27.057491 : ConnectionStateChange event: ConnectionEvent.connected
Reason: null
2020-07-31 18:14:51.896204 : ConnectionStateChange event: ConnectionEvent.disconnected
Reason: ErrorInfo message=Application lEu0Vg disabled. (See https://help.ably.io/error/40300 for help.) code=40300 statusCode=403 href=https://help.ably.io/error/40300
2020-07-31 18:14:52.612866 : ConnectionStateChange event: ConnectionEvent.connecting
Reason: null
2020-07-31 18:14:52.926823 : ConnectionStateChange event: ConnectionEvent.failed
Reason: ErrorInfo message=Application lEu0Vg disabled. (See https://help.ably.io/error/40300 for help.) code=40300 statusCode=403 href=https://help.ably.io/error/40300
```
result: As expected

#### Restricted API key ✅

Trying to attach to a channel with an API key with restricted access

```js
2020-07-31 19:40:59.586416 : ConnectionStateChange event: ConnectionEvent.connecting
Reason: null
2020-07-31 19:41:00.463867 : ConnectionStateChange event: ConnectionEvent.connected
Reason: null
//Here, trying to attach to channel that is not allowed with this API key
Attaching to channel test-channel: Current state ChannelState.initialized
ChannelStateChange: ChannelState.attaching
Reason: null
ChannelStateChange: ChannelState.suspended
Reason: ErrorInfo message=Channel denied access based on given capability; channelId = test-channel. (See https://help.ably.io/error/40160 for help.) code=40160 statusCode=401 href=https://help.ably.io/error/40160
Unable to attach to channel: ErrorInfo message=Channel denied access based on given capability; channelId = test-channel. (See https://help.ably.io/error/40160 for help.) code=40160 statusCode=401 href=https://help.ably.io/error/40160
```
result: As expected

#### Un-restricting the above API Key, and manually re-connecting ❓

The above, with updated API key settings to remove restrictions and trying a fresh connection again by `realtime.connect()` on same realtime instance:

```js
2020-07-31 19:45:58.041846 : ConnectionStateChange event: ConnectionEvent.connecting
Reason: null
2020-07-31 19:45:58.315391 : ConnectionStateChange event: ConnectionEvent.connected
Reason: ErrorInfo message=Connection unavailable code=80002 statusCode=503 href=https://help.ably.io/error/80002
Attaching to channel test-channel: Current state ChannelState.suspended
```

result: **Unexpected behavior**

⚠️ ⚠️ ⚠️ ⚠️ ⚠️

reason:
`ConnectionState` changes to `connected` but with a reason **"Connection Unavailable"**.
This isn't expected


If a fresh realtime instance is used to connect, it connects as expected.

#### Another observation ❗

If a channel is already connected, it can publish and subscribe to/from that channel
even if API Key settings are updated with new restricted permissions.

### RTN15a ✅

Token has expired => re-authenticates automatically. Calls authCallback if configured

### RTN15g (RTN15g1, RTN15g2, RTN15g3) ✅

These items are closely related with logs shared under heading `RTN14c, RTN14d, RTN14e, RTN14f` above

### RTN15i, RTN15b (RTN15b1, RTN15b2), RTN15c (RTN15c1, RTN15c2, RTN15c3, RTN15c4), RTN15e ⛔

Ignoring as these spec items for now as they revolve around `ProtocolMessage`

### RTN15f ⛔

Ignoring as these spec items for now as they revolve around `ACK` and `NACK`

### RTN15d ⛔

Integration Tests - unsure of the priority on integration tests at the moment.
Original file line number Diff line number Diff line change
Expand Up @@ -227,10 +227,10 @@ private ClientOptions decodeClientOptions(Map<String, Object> jsonMap) {
readValueFromJson(jsonMap, PlatformConstants.TxClientOptions.httpOpenTimeout, v -> o.httpOpenTimeout = (Integer) v);
readValueFromJson(jsonMap, PlatformConstants.TxClientOptions.httpRequestTimeout, v -> o.httpRequestTimeout = (Integer) v);
readValueFromJson(jsonMap, PlatformConstants.TxClientOptions.httpMaxRetryCount, v -> o.httpMaxRetryCount = (Integer) v);
readValueFromJson(jsonMap, PlatformConstants.TxClientOptions.realtimeRequestTimeout, v -> o.realtimeRequestTimeout = (Long) v);
readValueFromJson(jsonMap, PlatformConstants.TxClientOptions.realtimeRequestTimeout, v -> o.realtimeRequestTimeout = readValueAsLong(v));
readValueFromJson(jsonMap, PlatformConstants.TxClientOptions.fallbackHosts, v -> o.fallbackHosts = (String[]) v);
readValueFromJson(jsonMap, PlatformConstants.TxClientOptions.fallbackHostsUseDefault, v -> o.fallbackHostsUseDefault = (Boolean) v);
readValueFromJson(jsonMap, PlatformConstants.TxClientOptions.fallbackRetryTimeout, v -> o.fallbackRetryTimeout = (Long) v);
readValueFromJson(jsonMap, PlatformConstants.TxClientOptions.fallbackRetryTimeout, v -> o.fallbackRetryTimeout = readValueAsLong(v));
readValueFromJson(jsonMap, PlatformConstants.TxClientOptions.defaultTokenParams, v -> o.defaultTokenParams = readTokenParams((Map<String, Object>) v));
readValueFromJson(jsonMap, PlatformConstants.TxClientOptions.channelRetryTimeout, v -> o.channelRetryTimeout = (Integer) v);
readValueFromJson(jsonMap, PlatformConstants.TxClientOptions.transportParams, v -> o.transportParams = (Param[]) v);
Expand Down
Loading