Foundational changes in the protocol

While MQTT 5 is a major update to the existing protocol specification, the new version of the lightweight IoT protocol is more of an evolution rather than a revolution and retained all characteristics that contributed to its success: Its lightweightness, push communication, unique features, ease of use, extreme scalability, suitability for mobile networks and decoupling of communication participants.

Although some foundational mechanics were added or changed slightly, the new version still feels like MQTT and it sticks to its principles that made it the most popular Internet of Things protocol to date. This blog post will analyze everything you need to know about the foundational changes in version 5 of the MQTT specification before digging deep into the details of the new features during the next weeks.

At the end of this blog post, we have a video on the same topic that complements this blog post.

MQTT is still MQTT. Mostly.

The good news: If you’re familiar with MQTT 3.1.1 (if you aren’t, we recommend reading the MQTT Essentials series first!), then all principles and features you know about MQTT are still valid for MQTT v5. Some details of former features like Last Will and Testament changed a bit or some features were extended and additional popular features implemented by HiveMQ like TTL or Shared Subscriptions were added to the new specification.

The protocol also slightly changed on the wire and an additional control packet (AUTH) was added. But all in all, the MQTT version 5 is still clearly recognizable as MQTT.

Properties in the MQTT Header & Reason Codes

One of the most exciting and most flexible new MQTT 5 features is the possibility to add custom key-value properties in the MQTT header. This feature deserves its own blog post as it is a game changer for many deployments. Similar to protocols like HTTP, MQTT clients and brokers can add an arbitrary number of custom (or pre-defined) headers to carry metadata. This metadata can be used for application specific data. Pre-defined headers are used for the implementation of most of the new MQTT features.

Many MQTT packets now also include Reason Codes. A Reason Code indicates that a pre-defined protocol error occurred. These reason codes are typically carried on acknowledgement packets and allow client and broker to interpret error conditions (and potentially work around them). The Reason Codes are sometimes called Negative Acknowledgements. The following MQTT packets can carry Reason Codes:

• CONNACK
• PUBACK
• PUBREC
• PUBREL
• PUBCOMP
• SUBACK
• UNSUBACK
• AUTH
• DISCONNECT

Reason Codes for negative acknowledgements range from “Quota Exceeded” to “Protocol Error”. Clients and brokers are responsible for interpreting these new Reason Codes.

CONNACK Return Codes for unsupported features

With the popularity of MQTT, a lot of MQTT implementations were created and offered by companies. Not all of these implementations are completely MQTT specification compatible since sometimes features are not implemented like QoS 2, retained messages or persistent sessions. On a side note, HiveMQ is of course fully MQTT specification conformal and supports all features.

MQTT 5 provides a way for incomplete MQTT implementations (as often found in SaaS offerings) to indicate that the broker does not support specific features. It’s the client’s job to make sure that none of the unsupported features are used. The broker implementation uses pre-defined headers in the CONNACK packet (which is sent by the broker after the client sent a CONNECT packet) to indicate that specific features are not supported. These headers can of course also be used to send a notice to the client that it has no permission to use specific features.

The following pre-defined headers for indicating unimplemented features (or features not permitted for use by the client) are available in MQTT v5:

These return codes are a major step forward for communicating the permissions of individual MQTT clients in heterogeneous environments. The downside of this new functionality is that MQTT clients need to implement the interpretation of these codes themselves and need to make sure that application programmers don’t use features that are not supported by the broker (or the client has no permission for). HiveMQ supports all MQTT 5 features 100%, so these custom headers are only expected to be used if the administrator choses to use them for permissions in deployments.

Clean Session is now Clean Start

A popular MQTT 3.1.1 functionality is the use of clean sessions by MQTT clients, which have temporary connections or don’t subscribe to messages at all. When connecting to the broker, the client had to choose to send a CONNECT packet with the cleanSession flag enabled or disabled. With a clean session, an MQTT client indicates that the broker should discard any data for the client as soon as the underlying TCP connection breaks or if the client decides to disconnect from the broker. Also, if there was a previous session associated with the client identifier on the broker, a cleanSession CONNECT packet forced the broker to delete the previous data.

With MQTT v5, a client can choose to use a Clean Start (indicated by the Clean Start flag in the CONNECT message). When using this flag, the broker discards any previous session data and the client starts with a fresh session. The session won’t be cleaned automatically after the TCP connection was closed between client and server. To trigger the deletion of the session after the client disconnected, a new header field called “Session Expiry Interval” must be set to the value 0.

The new Clean Start streamlines and simplifies the session handling of MQTT, as it allows more flexibility and is easier to implement than the cleanSession/persistent session concept. With MQTT 5 all session are persistent unless the “Session Expiry Interval” is 0. Deletion of a session occurs after the timeout or when the client reconnects with Clean Start.

Additional MQTT Packet

MQTT 5 introduces a new MQTT Packet: The AUTH packet. This new packet is extremely useful for implementing non-trivial authentication mechanisms and we expect that this packet will be used a lot in production environments. The exact semantics are covered in a future blog post.

For the moment, it’s important to realize that this new packet can be sent by brokers and clients after connection establishment to use complex challenge/response authentication methods (like SCRAM or Kerberos as defined in the SASL framework), but can also be used for state-of-the-art authentication methods for IoT like OAuth. This packet also allows re-authentication of MQTT clients without closing the connection. Stay tuned for a deep dive into the packet very soon!

New Data Type: UTF-8 String pairs

The advent of custom headers also required a new data type to be introduced. UTF-8 string pairs. This string pair is essentially a key-value structure with both, key and value, as String data type. This data type is currently only used for custom headers.

With this new data type, MQTT uses 7 different data types that are used on the wire:

• Bit
• Two Byte Integer
• Four Byte Integer
• UTF-8 Encoded String
• Variable Byte Integer
• Binary Data
• UTF-8 String Pair

Most application users typically use Binary Data and UTF-8 encoded Strings in the APIs of their MQTT library. With MQTT 5, UTF-8 String Pairs may also be used frequently. All other data types are hidden from the user but are used on the wire to craft valid MQTT packets by the MQTT client libraries and brokers.

Bi-directional DISCONNECT packets

With MQTT 3.1.1, the client could indicate that it wanted to disconnect gracefully by sending a DISCONNECT packet prior to closing the underlying TCP connection. There was no way for the MQTT broker to notify a MQTT client that something bad happened and that the broker is going to close the TCP connection. This changed with the new protocol version. The broker is now allowed to send a MQTT DISCONNECT packet prior to closing the socket. The client is now able to interpret the reason why it was disconnected and take the respective action. A broker is not required to indicate the exact reason (e.g. for security reasons), but at least for developing applications this helps a lot to figure out why a connection was closed by the broker.

Of course DISCONNECT packets can carry Reason Codes, so it’s easy to indicate what was the reason for the disconnection (e.g. in case of invalid permissions).

No retry for QoS 1 and 2 messages

MQTT clients use standing TCP (or similar protocols with the same guarantees) connections as underlying transport. A healthy TCP connection gives bi-directional connectivity with exactly-once and in-order guarantees, so all MQTT packets sent by clients or brokers will arrive on the other end. In case the TCP connection breaks, while the message is in-flight, QoS 1 and 2 give message delivery guarantees over multiple TCP connections.

MQTT 3.1.1 allowed the re-delivery of MQTT messages while the TCP connection is healthy. In practice, this is a very bad idea, since overloaded MQTT clients may get overloaded even more. Just imagine a case where a MQTT client receives a message from a MQTT broker and needs 11 seconds to process the message (and would acknowledge the packet after the processing). Now imagine the broker would retransmit the message after a 10 second timeout. There is no advantage to this approach and it just uses precious bandwidth and overloads the MQTT client even more.

With MQTT 5, brokers and clients are not allowed to retransmit MQTT messages for healthy TCP connections. The brokers and clients must re-send unacknowledged packets when the TCP connection was closed, though. So the QoS 1 and 2 guarantees are as important as with MQTT 3.1.1.

In case you rely on retransmission packets in your use case (e.g. because your implementation does not acknowledge packets in certain cases), we suggest reconsidering this decision before upgrading to MQTT v5.

Using passwords without usernames

MQTT 3.1.1 required the MQTT client to send a username when using a password in the CONNECT packet. For certain use cases this was very inconvenient in case there was no username. A good example would be the use of OAuth, which uses a JSON web token as the only authentication and authorization information. When using such a token with MQTT 3.1.1, static usernames were used frequently, as the only relevant information was in the password field.

While there are more elegant ways in MQTT 5 to carry tokens (e.g. via the AUTH packet), it is still possible to utilize the password field of the CONNECT packet. Now users can just use the password field and don’t need to fill out the username anymore.

Conclusion

Although the MQTT protocol stays basically the same, there are some small changes under the hood, which are enablers for many of the new features in version 5 of the popular IoT protocol. As a user of MQTT libraries, most of these changes are good to know but do not affect how MQTT is used. Developers for MQTT libraries and brokers need to take care of the changes, especially on the changes on the protocol nitty-grittys. There were also some small changes in the specified behavior (e.g. retransmission of messages), so it’s good to revisit design decisions of deployments when upgrading to MQTT 5.

What do you like most of the foundational changes in MQTT? Let us know in the comments! Do check out MQTT Glossary to know all the key MQTT5 terminologies.

Learn more about MQTT5

Over the next few weeks, we are going to take a technical deep dive into the new MQTT 5 features. In Part 3 of this series, we will tell you why you should upgrade now!

Have an awesome week, The HiveMQ Team

About The HiveMQ Team

We love writing about MQTT, IoT protocols and architecture in general. Our experts are here to help, so reach out to us if we can help!
Contact us