This document aims to list our all the inputs from how some features are tested manually and feasible tests can be converted to automated tests. This document currently targets **[RTN14](https://docs.ably.com/client-lib-development-guide/features/#RTN14)** and **[RTN15](https://docs.ably.com/client-lib-development-guide/features/#RTN15)** sections of the spec, partially ## [RTN14](https://docs.ably.com/client-lib-development-guide/features/#RTN14) ### RTN14a ✅ **Actions performed** 1. Create realtime using an Invalid API Key 2. attach `connectionStateChange` listeners and `channelStateChange` listeners 3. connect to realtime instance using `realtime.connect()` Logs ```js 2020-07-31 17:31:29.424647 : ConnectionStateChange event: ConnectionEvent.connecting Reason: null 2020-07-31 17:31:30.115235 : ConnectionStateChange event: ConnectionEvent.failed Reason: ErrorInfo message=Invalid key in request: asdf_d231:fawefsd-afwesd. (See https://help.ably.io/error/40005 for help.) code=40005 statusCode=400 href=https://help.ably.io/error/40005 ``` Result: As expected ### RTN14b, RTN14g ⛔ Ignoring as these spec items for now as they revolve around `ProtocolMessage` ### RTN14c, RTN14d, RTN14e, RTN14f ✅ **Turning wifi off** - reconnect triggers every 15 seconds loops between states: `connecting` and `disconnected` every 15s for 60s (i.e., until the default `connectionStateTtl`) there-further, state changes between `connecting` and `suspended` continuously. logs ```js 2020-07-31 16:44:48.912345 : ConnectionStateChange event: ConnectionEvent.connecting Reason: null 2020-07-31 16:44:48.965277 : ConnectionStateChange event: ConnectionEvent.disconnected Reason: ErrorInfo message=sandbox-realtime.ably.io code=80000 statusCode=503 href=https://help.ably.io/error/80000 2020-07-31 16:45:03.950481 : ConnectionStateChange event: ConnectionEvent.connecting Reason: null 2020-07-31 16:45:04.014063 : ConnectionStateChange event: ConnectionEvent.disconnected Reason: ErrorInfo message=sandbox-realtime.ably.io code=80000 statusCode=503 href=https://help.ably.io/error/80000 2020-07-31 16:45:18.988102 : ConnectionStateChange event: ConnectionEvent.connecting Reason: null 2020-07-31 16:45:19.062352 : ConnectionStateChange event: ConnectionEvent.disconnected Reason: ErrorInfo message=sandbox-realtime.ably.io code=80000 statusCode=503 href=https://help.ably.io/error/80000 2020-07-31 16:45:34.020740 : ConnectionStateChange event: ConnectionEvent.connecting Reason: null 2020-07-31 16:45:34.109471 : ConnectionStateChange event: ConnectionEvent.disconnected Reason: ErrorInfo message=sandbox-realtime.ably.io code=80000 statusCode=503 href=https://help.ably.io/error/80000 2020-07-31 16:45:49.056232 : ConnectionStateChange event: ConnectionEvent.connecting Reason: null 2020-07-31 16:45:49.127278 : ConnectionStateChange event: ConnectionEvent.suspended Reason: ErrorInfo message=sandbox-realtime.ably.io code=80000 statusCode=503 href=https://help.ably.io/error/80000 2020-07-31 16:46:49.101562 : ConnectionStateChange event: ConnectionEvent.connecting Reason: null 2020-07-31 16:46:49.178212 : ConnectionStateChange event: ConnectionEvent.suspended Reason: ErrorInfo message=sandbox-realtime.ably.io code=80000 statusCode=503 href=https://help.ably.io/error/80000 2020-07-31 16:47:49.139349 : ConnectionStateChange event: ConnectionEvent.connecting Reason: null 2020-07-31 16:47:49.213962 : ConnectionStateChange event: ConnectionEvent.suspended Reason: ErrorInfo message=sandbox-realtime.ably.io code=80000 statusCode=503 href=https://help.ably.io/error/80000 ``` result: As expected ## [RTN15](https://docs.ably.com/client-lib-development-guide/features/#RTN15) ### RTN15h, RTN15h1, RTN15h2 ✅ #### Temporary key ✅ Created a temporary key on web console and deleted that after connection is established. Connection disconnected, and eventually status changed to failed after 1 retry. ```js 2020-07-31 18:25:05.070597 : ConnectionStateChange event: ConnectionEvent.connecting Reason: null 2020-07-31 18:25:05.296538 : ConnectionStateChange event: ConnectionEvent.connected Reason: null 2020-07-31 18:25:12.123308 : ConnectionStateChange event: ConnectionEvent.disconnected Reason: ErrorInfo message=Key/token status changed (revoke). (See https://help.ably.io/error/40131 for help.) code=40131 statusCode=401 href=https://help.ably.io/error/40131 2020-07-31 18:25:12.126130 : ConnectionStateChange event: ConnectionEvent.connecting Reason: null 2020-07-31 18:25:12.408257 : ConnectionStateChange event: ConnectionEvent.failed Reason: ErrorInfo message=Key revoked. (See https://help.ably.io/error/40131 for help.) code=40131 statusCode=401 href=https://help.ably.io/error/40131 ``` result: As expected #### Temporary app ✅ Created a temporary app on web console and deleted that after connection is established. Connection disconnected, and eventually status changed to failed after 1 retry. ```js 2020-07-31 18:14:26.818151 : ConnectionStateChange event: ConnectionEvent.connecting Reason: null 2020-07-31 18:14:27.057491 : ConnectionStateChange event: ConnectionEvent.connected Reason: null 2020-07-31 18:14:51.896204 : ConnectionStateChange event: ConnectionEvent.disconnected Reason: ErrorInfo message=Application lEu0Vg disabled. (See https://help.ably.io/error/40300 for help.) code=40300 statusCode=403 href=https://help.ably.io/error/40300 2020-07-31 18:14:52.612866 : ConnectionStateChange event: ConnectionEvent.connecting Reason: null 2020-07-31 18:14:52.926823 : ConnectionStateChange event: ConnectionEvent.failed Reason: ErrorInfo message=Application lEu0Vg disabled. (See https://help.ably.io/error/40300 for help.) code=40300 statusCode=403 href=https://help.ably.io/error/40300 ``` result: As expected #### Restricted API key ✅ Trying to attach to a channel with an API key with restricted access ```js 2020-07-31 19:40:59.586416 : ConnectionStateChange event: ConnectionEvent.connecting Reason: null 2020-07-31 19:41:00.463867 : ConnectionStateChange event: ConnectionEvent.connected Reason: null //Here, trying to attach to channel that is not allowed with this API key Attaching to channel test-channel: Current state ChannelState.initialized ChannelStateChange: ChannelState.attaching Reason: null ChannelStateChange: ChannelState.suspended Reason: ErrorInfo message=Channel denied access based on given capability; channelId = test-channel. (See https://help.ably.io/error/40160 for help.) code=40160 statusCode=401 href=https://help.ably.io/error/40160 Unable to attach to channel: ErrorInfo message=Channel denied access based on given capability; channelId = test-channel. (See https://help.ably.io/error/40160 for help.) code=40160 statusCode=401 href=https://help.ably.io/error/40160 ``` result: As expected #### Un-restricting the above API Key, and manually re-connecting ❓ The above, with updated API key settings to remove restrictions and trying a fresh connection again by `realtime.connect()` on same realtime instance: ```js 2020-07-31 19:45:58.041846 : ConnectionStateChange event: ConnectionEvent.connecting Reason: null 2020-07-31 19:45:58.315391 : ConnectionStateChange event: ConnectionEvent.connected Reason: ErrorInfo message=Connection unavailable code=80002 statusCode=503 href=https://help.ably.io/error/80002 Attaching to channel test-channel: Current state ChannelState.suspended ``` result: **Unexpected behavior** ⚠️ ⚠️ ⚠️ ⚠️ ⚠️ reason: `ConnectionState` changes to `connected` but with a reason **"Connection Unavailable"**. This isn't expected If a fresh realtime instance is used to connect, it connects as expected. #### Another observation ❗ If a channel is already connected, it can publish and subscribe to/from that channel even if API Key settings are updated with new restricted permissions. ### RTN15a ✅ Token has expired => re-authenticates automatically. Calls authCallback if configured ### RTN15g (RTN15g1, RTN15g2, RTN15g3) ✅ These items are closely related with logs shared under heading `RTN14c, RTN14d, RTN14e, RTN14f` above ### RTN15i, RTN15b (RTN15b1, RTN15b2), RTN15c (RTN15c1, RTN15c2, RTN15c3, RTN15c4), RTN15e ⛔ Ignoring as these spec items for now as they revolve around `ProtocolMessage` ### RTN15f ⛔ Ignoring as these spec items for now as they revolve around `ACK` and `NACK` ### RTN15d ⛔ Integration Tests - unsure of the priority on integration tests at the moment.