Foundational changes in the protocol

Welcome to Part 2 of MQTT 5 Essentials Series. In Part 1 of MQTT 5 Essentials series, Introduction to MQTT 5 Protocol, we explored the fundamentals of MQTT, discussing its origin, importance in the IoT space, and provided an overview of its basic concepts and characteristics. In Part 2, we will dive more deeply into the changes introduced in MQTT 5, how they enhance the effectiveness of this IoT protocol, and closely cover MQTT 5 vs. MQTT 3.

While MQTT 5 presents a substantial update to MQTT 3.1.1, it remains more of an evolution than a revolution, faithfully retaining all the features that have contributed to its success. Its lightweightness, push communication, unique attributes, ease of use, exceptional scalability, suitability for mobile networks, and decoupling of communication participants all endure in this latest version. However, several foundational mechanics have been added or slightly adjusted, ensuring that while the new version still has the feel of MQTT, it incorporates improvements that reinforce its standing as the most popular Internet of Things protocol to date.

In this article, we’ll break down everything you need to know about the foundational changes in version 5 of the MQTT specification. We’ll also lay the groundwork for upcoming articles, where we’ll dig deep into the details of the new features.

What Improvements are Seen in MQTT 5 Compared to MQTT 3?

The reassuring news for those acquainted with MQTT 3.1.1 is that the fundamental principles and features remain unchanged in MQTT V5. If you’re new to MQTT, we suggest starting with the MQTT Essentials series for a foundational understanding. Despite certain modifications and expansions, the heart of the MQTT you are familiar with persists in version 5. Noteworthy adjustments include slight changes to pre-existing features such as Last Will and Testament and the inclusion of popular HiveMQ features like TTL and Shared Subscriptions.

The underlying protocol has experienced minor transformations, and an additional control packet—AUTH—has been incorporated. However, these changes do not obscure MQTT’s core identity. In essence, MQTT v5 is still unambiguously MQTT, retaining its recognizable characteristics while enhancing its capabilities.

What Properties Are Available in MQTT 5 Header & Reason Codes?

Undoubtedly, one of the most transformative and flexible feature introduced in MQTT 5 is the ability to incorporate custom key-value properties in the MQTT header. This capability can potentially revolutionize many deployments, and is of such significance that it warrants an entire blog post dedicated to it. Like protocols such as HTTP, MQTT clients and brokers can add an unlimited number of custom or pre-defined headers to carry metadata. This metadata is adaptable and can be tailored to specific application data, with pre-defined headers predominantly employed in the execution of most new MQTT features.

Additionally, MQTT packets now include Reason Codes, signifying the occurrence of a pre-defined protocol error. Traditionally found on acknowledgment packets, these Reason Codes facilitate error interpretation and potentially aid in devising remedies by both clients and brokers.

Occasionally referred to as Negative Acknowledgements, these Reason Codes can be found on a range of MQTT packets, such as:

• CONNACK
• PUBACK
• PUBREC
• PUBREL
• PUBCOMP
• SUBACK
• UNSUBACK
• AUTH
• DISCONNECT

The range of Reason Codes for Negative Acknowledgements varies widely, spanning from “Quota Exceeded” to “Protocol Error”. It is the joint responsibility of clients and brokers to decode and comprehend these newly introduced Reason Codes, further enriching the overall MQTT experience.

Having established the custom key-value properties and Reason Codes, let’s explore how MQTT handles unsupported features and uses CONNACK return codes.

Responding to Unsupported Features with CONNACK Return Codes

As MQTT’s popularity surged, various MQTT implementations began to emerge, proposed by a diverse range of companies. However, not all these implementations adhere strictly to the complete MQTT specification. Certain features, such as QoS 2, retained messages or persistent sessions may not consistently be implemented. In contrast, HiveMQ remains wholly conformed to the MQTT specification, comprehensively supporting all features. Pivoting toward the subject of unsupported features, MQTT 5 presents an ingenious resolution. It enables those implementations that are not entirely complete, often seen in SaaS offerings, to indicate the broker’s inability to support certain features. The onus is then placed upon the client to ensure these unsupported features are not utilized. The broker employs pre-defined headers in the CONNACK packet, sent in response to the client’s CONNECT packet, to communicate the lack of support for specific features. These headers can also be leveraged to notify the client that they are not permitted to use certain features.

What are the Pre-Defined Headers Available in MQTT 5?

Here are the pre-defined headers available in MQTT v5 for indicating unimplemented features or features not authorized for client use:

These return codes represent a significant stride forward in expressing the permissions of individual MQTT clients. However, this new capability carries an inherent trade-off; MQTT clients must implement the interpretation of these codes independently, and must ensure that application developers avoid using features unsupported by the broker, or those the client lacks permission for.

As a note of assurance, HiveMQ is fully compliant with all MQTT 5 features, ensuring these custom headers would only be utilized at the administrator’s discretion for setting permissions in deployments.

Enhanced Session Management: From Clean Session to Clean Start in MQTT 5

The concept of “Clean Session,” a prominent feature in MQTT 3.1.1, is now succeeded by “Clean Start” in MQTT v5. The Clean Session feature was highly used in MQTT 3.1.1 by clients that either had temporary connections or didn’t subscribe to messages. Upon connecting to the broker, the client was required to send a CONNECT packet with either the Clean Session flag enabled or disabled. If enabled, it indicated to the broker that all client data should be discarded when the underlying TCP connection was severed or upon the client’s decision to disconnect from the broker. Moreover, if a previous session was associated with the client identifier on the broker, a Clean Session CONNECT packet compelled the broker to discard the prior data.

MQTT 5 introduces the Clean Start option, signaled by the Clean Start flag in the CONNECT message. With this flag, the broker dismisses any prior session data, and the client initiates a new session. However, the session isn’t automatically cleared when the TCP connection is closed between the client and the server. To prompt the deletion of the session post-disconnection, a new header field, Session Expiry Interval, must be set to 0.

This revised Clean Start functionality enhances and simplifies MQTT’s session handling, offering more flexibility and easier implementation compared to the former Clean Session/persistent session concept. In MQTT 5, all sessions persist unless the “Session Expiry Interval” is set to 0. Session deletion takes place either after the interval timeout or when the client reconnects using Clean Start.

What is AUTH Packet in MQTT 5?

In an exciting development, MQTT 5 brings forth a new packet type: the AUTH packet. Serving as a vital tool in implementing non-trivial authentication mechanisms, we anticipate this packet to be integral in production environments. We will delve into its exact semantics in another article.

It’s crucial to understand that this novel packet can be dispatched by both brokers and clients post-connection establishment. It enables the use of intricate challenge/response authentication methods, such as SCRAM or Kerberos as outlined in the SASL framework and is also compatible with cutting-edge IoT authentication methods like OAuth. Importantly, the AUTH packet allows for the re-authentication of MQTT clients without requiring the termination of the connection.

MQTT 5: Enriching MQTT with UTF-8 String Pairs

To accommodate the custom headers, a new data type was necessitated, and thus, the UTF-8 string pairs were introduced. Essentially, a string pair is a key-value structure wherein both the key and value are of the String data type. At present, this data type is primarily used for custom headers.

This novel addition expands the spectrum of MQTT data types utilized on the wire to a total of seven:

1. Bit
2. Two Byte Integer
3. Four Byte Integer
4. UTF-8 Encoded String
5. Variable Byte Integer
6. Binary Data
7. UTF-8 String Pair

For most application users, Binary Data and UTF-8 Encoded Strings remain the go-to data types within their MQTT library APIs. However, with the onset of MQTT 5, the UTF-8 String Pairs are anticipated to gain frequent usage.

While the remaining data types might not be directly visible to users, they are leveraged on the wire to construct valid MQTT packets by MQTT client libraries and brokers.

How Does MQTT 5 Enhance Communication with Bi-directional DISCONNECT Packets?

Under the MQTT 3.1.1 framework, clients could gracefully terminate their connection by dispatching a DISCONNECT packet before closing the underlying TCP connection. However, there was no provision for the MQTT broker to inform the client of a potential issue that necessitates closing the TCP connection. This shortcoming has been addressed in the new protocol version.

In the enhanced MQTT, the broker is authorized to transmit a DISCONNECT packet before severing the socket connection. This provision empowers the client to understand the cause behind the disconnection and devise a suitable response accordingly. While the broker is not mandated to disclose the exact reason (for security purposes, for instance), this feature benefits developers as it provides valuable insight into why the broker terminated a connection.

Another valuable addition is that DISCONNECT packets can now carry Reason Codes, simplifying the process of revealing the disconnection’s rationale, such as in the case of invalid permissions.

Revamping QoS 1 and 2 in MQTT 5: Eliminating Retries for Healthy Connections

MQTT employs persistent TCP connections (or similar protocols with identical assurances) for the underlying transport. A robust TCP connection ensures bi-directional connectivity with exactly-once and in-order guarantees, meaning all MQTT packets dispatched by clients or brokers will be received at the other end. When the TCP connection is disrupted while the message is in transit, QoS 1 and 2 ensure message delivery across multiple TCP connections.

Under the MQTT 3.1.1 protocol, re-delivery of MQTT messages was permissible even when the TCP connection was healthy. This often proved detrimental in practice, risking overloading already burdened MQTT clients. Consider a scenario where an MQTT client takes 11 seconds to process a message received from a broker to acknowledge the packet after processing. There is no tangible advantage if the broker retransmits the message after a 10-second timeout. This approach merely consumes valuable bandwidth and further overwhelms the MQTT client.

With the advent of MQTT 5, retransmission of MQTT messages for healthy TCP connections is disallowed for both brokers and clients. However, brokers and clients must resend unacknowledged packets when the TCP connection has been severed. Thus, the QoS 1 and 2 guarantees remain as vital as they were in MQTT 3.1.1.

If your use case depends on retransmission packets (for instance, if your implementation fails to acknowledge packets under certain circumstances), we recommend reevaluating this strategy before upgrading to MQTT v5.

How Does MQTT 5 Simplify Authentication?

In the MQTT 3.1.1 protocol, MQTT clients needed to provide a username when utilizing a password in the CONNECT packet. This proved inconvenient for some use cases where a username was unnecessary. One prominent example includes using OAuth, which leverages a JSON web token for authentication and authorization information. With MQTT 3.1.1, static usernames were often utilized given that the critical information was contained within the password field.

With the advent of MQTT 5, the protocol has introduced more refined ways to handle tokens, for instance, via the AUTH packet. However, using the password field of the CONNECT packet is still feasible. The major improvement here is that users can simply leverage the password field without the obligation to fill out the username. This adjustment offers a more streamlined and straightforward approach to authentication in specific scenarios.

Wrapping Up: Unveiling the Foundations of MQTT 5

While the core of the MQTT protocol remains relatively unchanged, subtle refinements have been implemented under the surface, laying the groundwork for many new features in version 5 of this widely adopted IoT protocol. As a user leveraging MQTT libraries, these modifications may seem minor and do not radically alter the way MQTT is utilized. However, for developers working on MQTT libraries and brokers, these changes, particularly those related to protocol nuances, are critical and demand attention.

Some shifts in specified behavior, such as message retransmission, necessitate revisiting deployment design decisions during the transition to MQTT 5.

In Part 3 of this MQTT 5 Essentials series, we will cover seven reasons to upgrade to it from MQTT 3.1.1.

Sign up for our newsletter to get regular updates. Subscribe to our RSS feed here to stay updated. We encourage you to visit our MQTT Glossary for an in-depth understanding of the essential MQTT terminologies. It will equip you with the necessary vocabulary to grasp the complexities of MQTT and its various versions. Watch the video below that complements the concepts discussed in this article.

FAQs on MQTT5 Vs. MQTT 3

About HiveMQ Team

We love writing about MQTT, IoT protocols and architecture in general. Our experts are here to help, so reach out to us if we can help!

Contact HiveMQ