Connecting to AWS IoT broker (without Websocket)

[PaulFake]

When I try to connect to an AWS IoT broker, I get an MqttCommunicationClosedGracefullyException. Here is a trace:

Trying to connect with server (<endpoint>.iot.us-west-2.amazonaws.com:8883).
Connecting [Timeout=00:00:10]
Connection with server established.
Start receiving packets.
TX >>> Connect: [ClientId=<clientId>] [Username=] [Password=] [KeepAlivePeriod=30] [CleanSession=True]
MQTT communication exception while receiving packets. (MQTTnet.Exceptions.MqttCommunicationClosedGracefullyException: Exception of type 'MQTTnet.Exceptions.MqttCommunicationClosedGracefullyException' was thrown.)
   at MQTTnet.Serializer.MqttPacketReader.ReadFixedHeaderAsync(IMqttChannel channel, Byte[] fixedHeaderBuffer, Byte[] singleByteBuffer, CancellationToken cancellationToken)
   at MQTTnet.Adapter.MqttChannelAdapter.ReceiveAsync(CancellationToken cancellationToken)
   at MQTTnet.Adapter.MqttChannelAdapter.ReceivePacketAsync(TimeSpan timeout, CancellationToken cancellationToken)
   at MQTTnet.Client.MqttClient.ReceivePacketsAsync(CancellationToken cancellationToken)
Error while connecting with server. (MQTTnet.Exceptions.MqttCommunicationClosedGracefullyException: Exception of type 'MQTTnet.Exceptions.MqttCommunicationClosedGracefullyException' was thrown.)
   at MQTTnet.Serializer.MqttPacketReader.ReadFixedHeaderAsync(IMqttChannel channel, Byte[] fixedHeaderBuffer, Byte[] singleByteBuffer, CancellationToken cancellationToken)
   at MQTTnet.Adapter.MqttChannelAdapter.ReceiveAsync(CancellationToken cancellationToken)
   at MQTTnet.Adapter.MqttChannelAdapter.ReceivePacketAsync(TimeSpan timeout, CancellationToken cancellationToken)
   at MQTTnet.Client.MqttClient.ReceivePacketsAsync(CancellationToken cancellationToken)
   at MQTTnet.Internal.TaskExtensions.TimeoutAfterAsync[TResult](Func`2 action, TimeSpan timeout, CancellationToken cancellationToken)
   at MQTTnet.Client.MqttClient.SendAndReceiveAsync[TResponsePacket](MqttBasePacket requestPacket, CancellationToken cancellationToken)
   at MQTTnet.Client.MqttClient.AuthenticateAsync(MqttApplicationMessage willApplicationMessage, CancellationToken cancellationToken)
   at MQTTnet.Client.MqttClient.ConnectAsync(IMqttClientOptions options)
Disconnecting [Timeout=00:00:10]
Disconnected from adapter.
Disconnected.
Stopped receiving packets.
Stopped

The server doesn't log anything useful, aside from a generic connection error ("status": "Failure", "eventType": "Connect"). I've ruled out TLS issues as the culprit, as I've been able to connect to the same server using the same certificates and key using M2MQTT and MQTT.fx. I've also been able to connect to other non-AWS brokers with MQTTnet without issue (and using TLS). This problem is unique to MQTTnet with AWS, and I can't help but think there's a bug somewhere.

[chkr1011]

Hi, the topic about MQTTnet and AWS is open since a while and some people are able to connect but they do not report what they need to do 😄 So for now I cannot fix the issue.
Best regards
Christian

[PaulFake]

Maybe this will help some people. My team's been digging for a few days, and while we haven't completely solved the issue, we've made some progress. The biggest thing is that AWS doesn't support QoS 2 or the retain flag. It will automatically close the connection if either of those are requested. Now, changing the QoS to 1 and setting retain to false allows us to connect with MQTTnet on some of our systems. Other systems will connect for a few millseconds and then disconnect. It's very odd, and the same thing happens with M2MQTT, but not with MQTT.fx. If we are able to solve the issue, I'll update this thread and hopefully help some others who are struggling with the same thing.

So, in short, I was too quick to suggest it could be a bug in MQTTnet :)

[morganics]

Yep - I also have the same issue, with the same trace (I didn't try alternative clients). What is required to report the issue?

[jgmdavies]

Same issues here - any updates / advice from anyone please?

[PaulFake]

Setting retain to false and QoS to a maximum of 1 doesn't work for either of you? I've been able to connect to AWS just fine since Oct 23, 2018.

[jgmdavies]

Thanks Paul.
I'm not an MQTTnet expert, but I assume you mean using those options on a call to PublishAsync?
In my case the MqttCommunicationClosedGracefullyException is happening during the Connect, before any publish.
Jim

[PaulFake]

Hm, yeah I was thinking about publishing rather than connecting. I looked over my original post here and honestly I don't remember how I fixed the connection issue. Our system would publish immediately after connecting though, so maybe it really was just a publishing issue for me all along. I wish I could be more helpful. The only other thing I can remember getting hung up on with AWS was encryption. Are you providing certificates?

[jgmdavies]

Thanks Paul - sorry for long delay replying (doing 'other things').
I'm certainly providing certs, and I thought I'd got past those issues!
I'll revisit soon...
Jim

[jminnihan]

It would be a very helpful if the wiki presented some documentation on how to connect up to the AWS IoT Broker without web sockets with version 3.0.x of MQTTnet. BTW, There are some examples provided by the AWS team on how to do this using web sockets, but they also use the older 2.8.5 release of MQTTnet to do this, and they too provide no examples using 3.0.x of MQTTnet.

[SeppPenner]

BTW, There are some examples provided by the AWS team on how to do this using web sockets, but they also use the older 2.8.5 release of MQTTnet to do this, and they too provide no examples using 3.0.x of MQTTnet.

Do you have some links for me, @jminnihan? I would like to add this to the wiki.

[x37v]

Hm, yeah I was thinking about publishing rather than connecting. I looked over my original post here and honestly I don't remember how I fixed the connection issue. Our system would publish immediately after connecting though, so maybe it really was just a publishing issue for me all along. I wish I could be more helpful. The only other thing I can remember getting hung up on with AWS was encryption. Are you providing certificates?

@PaulFake do you know/remember what version of MQTTnet you were using?

[x37v]

Hm, yeah I was thinking about publishing rather than connecting. I looked over my original post here and honestly I don't remember how I fixed the connection issue. Our system would publish immediately after connecting though, so maybe it really was just a publishing issue for me all along. I wish I could be more helpful. The only other thing I can remember getting hung up on with AWS was encryption. Are you providing certificates?

@PaulFake do you know/remember what version of MQTTnet you were using?

also, what OS and .Net runtime version?

[x37v]

I was just able to publish to AWS IoT using a MQTTnet2.8.5 with NetCore 2.1 on a Mac, plain sockets, not web sockets.
I was also able to publish with the same setup except from inside an AWS labmda (linux?).

NetCore 2.0 did not work.

The Retained flag is what was killing me.. turned out I had a last will message with a retained flag set to true.
Also, I explicitly set the version to .WithProtocolVersion(MqttProtocolVersion.V311);

My connection does get dropped pretty often but I am able to publish. Maybe I need to change my keepalive.

[PaulFake]

@x37v Sorry for the long delay - I don't regularly sign into GitHub and see my messages. I believe we were using 2.8.5 with .NET Core 2.0, and I think it happened on both Win7 and Win10 systems.

[SeppPenner]

There is now a short documentation under https://github.com/chkr1011/MQTTnet/wiki/Client#connecting-with-amazon-aws and the library version is 3.0.8 right now... I will close this issue now. Feel free to contact us again if this still occurs and I will re-open it.