In the past, we’ve talked about the foundational changes that MQTT, the de facto standard protocol of the Internet of Things, introduced in version 5. We also looked at the top reasons why MQTT users upgrade to MQTT v5. Now, we want to examine each new feature in the MQTT 5 protocol, individually. Today’s post examines two very similar features: the Session Expiry Interval and the Message Expiry Interval.

Be sure to watch our video on the same topic at the end of this blog post.

How do Session Expiry Interval and Message Expiry Interval in MQTT 5 Work?

Let’s look at both expiry interval features separately.

Session Expiry Interval in MQTT 5

In the CONNECT packet, a connecting client can set a session expiry interval in seconds. This interval defines the period of time that the broker stores the session information of that particular MQTT client. When the session expiry interval is set to 0 or the CONNECT packet does not contain an expiry value, the session information is immediately removed from the broker as soon as the network connection of the client closes. The maximum session expiry interval is UINT_MAX (4,294,967,295), which makes the offline session last for slightly more than 136 years after the client disconnects.

Message Expiry Interval in MQTT 5

A client can set the message expiry interval in seconds for each PUBLISH message individually. This interval defines the period of time that the broker stores the PUBLISH message for any matching subscribers that are not currently connected. When no message expiry interval is set, the broker must store the message for matching subscribers indefinitely. When the retained=true option is set on the PUBLISH message, this interval also defines how long a message is retained on a topic.

Why Were the Expiry Intervals Introduced in MQTT 5?

Beyond understanding how these expiry intervals work, let’s take a closer look at why these features were introduced into the MQTT 5 specification and explore some practical uses.

As described in the introduction to our MQTT 5: Upgrade now. Here’s why. blog post: “After much deliberation, MQTT v5 became a standard in March 2019.” A big part of this deliberation within the OASIS committee (of which HiveMQ is a proud member) was spent gathering feedback, listening to long-term users of the previous MQTT 3.1 and 3.1.1 versions of the protocol, and figuring out how to advance the protocol with features that provide additional possibilities or increased simplicity for the user base.

MQTT 5’s Session Expiry Interval

The session expiry interval feature achieves both of these goals at the same time. Using earlier MQTT versions, the only way to remove the so called persistent sessions provided by the specification was to connect an MQTT client that uses the same client ID as the session you want to remove with a cleanSession=true flag. In scenarios where some of your IoT devices never reconnect, such as destruction or decommissioning of the devices or improperly cleaned up sessions that were left over from load testing environments, the session remnants can put unnecessary strain on a broker’s persistence. Enterprise ready MQTT brokers such as HiveMQ are equipped with additional administrative tools like the HiveMQ Control Center that make the management of unused sessions more convenient. However, you still need to know which session can actually be removed. The new session expiry interval feature lets you set a reasonable amount of time after which an unused session is automatically removed by the broker and frees up resources.

In addition to this automatic housekeeping functionality, the introduction of the session expiry interval has significantly simplified the handling of session states. Here are two diagrams, courtesy of Ian Craggs, that demonstrate the de-cluttering of the state transition.

De-cluttering of state transition in MQTT 3.1.1 and MQTT 5 (courtesy of Ian Craggs)

MQTT 5’s Message Expiry Interval

Similar to the session expiry interval, the primary motivation to add the message expiry interval to the MQTT protocol standard was to get an automated housekeeping mechanism. Many kinds of IoT devices, such as connected cars, are designed with the distinct possibility of prolonged periods without an internet connection. For these scenarios, MQTT provides persistent sessions and message queueing. Messages that are intended for the offline device are stored on the broker and delivered when the device regains a connection. In large scale deployments with hundreds of thousands or even millions of connected devices, you need to limit the offline message queue of each individual client. Frequently, the period that messages sent to IoT devices remain relevant varies significantly. Staying with the connected car example, information about the current traffic status is only relevant for a limited time. However, over-the-air firmware updates must be executed even if the car is offline for several weeks.

Setting an appropriate message expiry interval for messages that lose relevance within a certain amount of time and leaving continuously-relevant messages without an expiry ensures that broker resources for clients that are offline long term are used properly. This method also ensures that the clients don’t have to deal with large numbers of irrelevant messages when they reconnect. With retained messages, the message expiry interval can be used in the same way to ensure that retained messages are only delivered to new subscribers for a predefined time period.

GOTCHA: When the session for a client expires, all the messages that are queued for the client expire with the session, regardless of the individual message expiry status.

Summary and Additional Information

• Both the Session Expiry Interval and the Message Expiry Interval can be used to optimize resource management on the MQTT broker.
• As an aside: Expiry features were requested by many MQTT 3 users in the past. So much so that HiveMQ introduced session and message expiry as additional features in HiveMQ 3.3 (before MQTT 5 was released).
• The equivalent for cleanSession=true in the CONNECT packet of MQTTv3 in MQTTv5 is sessionExpiry=0 (or absent) and cleanStart=true
• The equivalent for cleanSession=false in the CONNECT packet of MQTTv3 in MQTTv5 is using a sessionExpire value that is greater than zero and cleanStart=false
• MQTT brokers like HiveMQ allow for a server side max value configuration for these expiry intervals. This is extremely useful in multi vendor projects, where the party operating the broker may not have control over the settings of the MQTT clients.

Learn More About MQTT 5

Over the next few weeks, we will continue our technical deep dive into all of the MQTT 5 features. In Part 5 of this series we will be talking about Improved client feedback & negative ACKs.

If you would like to join us, sign up for our newsletter to get regular updates and ensure that you don’t miss a post.

About Florian Raschbichler

Florian serves as the head of the HiveMQ support team with years of first hand experience overcoming challenges in achieving reliable, scalable, and secure IoT messaging for enterprise customers.
Contact Florian