Understanding MQTT Message Ordering

Understanding how MQTT handles message ordering is essential for building reliable and predictable IoT systems, especially when working with clustered environments or implementing standards like Eclipse Sparkplug. While the MQTT specification provides clear rules for message ordering within a topic on a single client connection, maintaining that order becomes more complex in distributed broker setups.

MQTT v3.1.1 & v5.0 Specifications

To provide context to the below discussion, here are the reference links: MQTT v3.1.1 specifications and MQTT v5.0 specifications. 

MQTT v3.1.1

Message Order within a Topic

MQTT 3.1.1 guarantees that for a single connection, the broker will deliver messages in the order they are received for each subscribed topic, respecting the QoS level:

“A Server MUST by default treat each Topic as an "Ordered Topic". It MAY provide an administrative or other mechanism to allow one or more Topics to be treated as an "Unordered Topic”.

When a Server processes a message that has been published to an Ordered Topic, it MUST follow the rules listed above when delivering messages to each of its subscribers. In addition it MUST send PUBLISH packets to consumers (for the same Topic and QoS) in the order that they were received from any given Client." (Page 56)

However, this does not apply across different topics or different connections.

Quoted from MQTT v3.1.1 Spec:

“A Client MUST follow these rules when implementing the protocol flows defined elsewhere in this chapter:

• When it re-sends any PUBLISH packets, it MUST re-send them in the order in which the original PUBLISH packets were sent (this applies to QoS 1 and QoS 2 messages)

• It MUST send PUBACK packets in the order in which the corresponding PUBLISH packets were received (QoS 1 messages)

• It MUST send PUBREC packets in the order in which the corresponding PUBLISH packets were received (QoS 2 messages)

• It MUST send PUBREL packets in the order in which the corresponding PUBREC packets were received (QoS 2 messages)."

Non-normative comment

The rules listed above ensure that when a stream of messages is published and subscribed to with QoS 1, the final copy of each message received by the subscribers will be in the order that they were originally published in, but the possibility of message duplication could result in a re-send of an earlier message being received after one of its successor messages. For example a publisher might send messages in the order 1,2,3,4 and the subscriber might receive them in the order 1,2,3,2,3,4.

If both Client and Server make sure that no more than one message is “in-flight” at any one time (by not sending a message until its predecessor has been acknowledged), then no QoS 1 message will be received after any later one. For example, a subscriber might receive them in the order 1,2,3,3,4 but not 1,2,3,2,3,4. Setting an in-flight window of 1 also means that order will be preserved even if the publisher sends a sequence of messages with different MQTT QoS levels on the same topic.”

Overlapping Subscriptions

If a client has multiple subscriptions that match a single topic (e.g., subscribing to both machines/devices/# and machines/devices/environment/temperator), it may receive the same message multiple times. The specification does not guarantee the delivery order of these duplicate messages relative to one another.

Quality of Service Level

• QoS 0 (At most once): Messages are delivered according to the best effort of the operating environment. No guarantees are made about delivery or order, and messages can be lost.

• QoS 1 (At least once): Messages are guaranteed to be delivered at least once but can arrive multiple times and out of order.

• QoS 2 (Exactly once): Messages are guaranteed to be delivered exactly once and in order. This level involves a handshake process to ensure that the message is delivered and acknowledged exactly once.

For more details about QoS in MQTT, read our blogs Quality of Service in MQTT: The Ultimate Guide and Debunking Common MQTT QoS Misconceptions.

MQTT v5.0

The fundamental specification for message ordering does not differ between MQTT v3.1.1 and MQTT v5.0. The core guarantee remains the same.

However, MQTT v5.0 introduces several features that create new considerations for message ordering beyond the guarantees provided in MQTT v3.1.1. While the core ordering rule for QoS 1 and 2 messages from a single publisher remains, features like Receive Maximium, Message Expiry, Shared Subscriptions and Request/Response pattern can intentionally alter or disrupt the sequence of messages a client receives.

The Role of Receive Maximum

The Receive Maximum was introduced as a new flow control mechanism in MQTT v5.0 where a client tells the broker the maximum number of unacknowledged published QoS 1 and QoS 2 messages it is willing to process at once (its "in-flight window"). By managing this window, the broker and client can more reliably handle the message sequence, preventing overload and ensuring the conditions for ordered delivery are met. By setting Receive Maximum to 1, a client can enforce strict in-order processing, as it will only handle one message at a time.

Message Expiry Intervals

This feature allows a publisher to set a lifetime for a message. The Message Expiry Interval is a property set on a PUBLISH packet. If the message is not delivered to matching subscribers within this interval, the broker will discard it. This is particularly relevant for messages with QoS 1 or 2 intended for offline clients with persistent sessions.

Impact on Ordering: Message expiry can intentionally create gaps in a message sequence. For example, imagine a client with a persistent session is offline. A publisher sends three messages (M1, M2, M3) in order to a topic the client is subscribed to. If M2 has a short expiry interval and expires before the client reconnects, the client will receive M1 and then M3. The ordered sequence is broken because the expired message (M2) was discarded by the broker.

Shared Subscriptions

Shared Subscriptions are a mechanism for distributing message load across multiple subscribers. Multiple clients subscribe to the same topic pattern (e.g., $share/ingest_pool_1/telemetry/vehicle_data) but receive messages in a round-robin or other load-balancing fashion. When a message is published to telemetry/vehicle_data, the broker delivers it to only one of the clients in the ingest_pool_1 share.

Impact on Ordering: Shared Subscriptions explicitly break the message ordering guarantee for the group of subscribers. While the broker receives messages from a publisher in a specific order, it distributes them among the subscribers. No single client in the shared group is guaranteed to receive all the messages, so it's impossible for any one of them to verify the complete order. This feature is used when processing throughput is more important than the strict ordering of messages for a single consumer.

For example, if messages M1, M2, M3, M4, M5, and M6  are published to group1 share containing Worker A, Worker B and Worker C, the distribution could be:

• Worker A receives M1 and M4

• Worker B receives M2 and M5

• Worker C receives M3 and M6

From Client A's perspective, the sequence is M1 -> M4, which is out of order and incomplete. This is a critical trade-off: you gain horizontal scalability and high availability at the cost of per-client message ordering. 

Request/Response Pattern

MQTT v5.0 formalizes the Request-Response pattern with two key properties: 

• Response Topic: The requester specifies a topic where it expects the response to be sent.

• Correlation Data: The requester includes data (like a unique request ID) that the responder must include in its response.

Impact on Ordering: This pattern isn't about preserving the FIFO order of a message stream but about maintaining a logical order for asynchronous conversations. A client can send multiple requests without waiting for each response. The Correlation Data allows the client to match an incoming response to its original request, regardless of the order in which the responses arrive.

For example, a client sends Request 1 and then Request 2. The responding service might process Request 2 faster and send Response 2 before Response 1. The requesting client can use the Correlation Data in each response to correctly handle them, even though they arrived "out of order" relative to when the requests were sent. This feature provides contextual ordering for parallel operations rather than strict stream ordering.

Strict Message Ordering in a Clustered MQTT Broker Setup

Strict ordering is inherently difficult in distributed systems without additional measures because they prioritize availability, scalability, no message loss, and fault tolerance over order guarantees. Strict message ordering on an MQTT topic is not guaranteed in a clustered MQTT broker setup. Here's why:

• Message Distribution Across Nodes:

  MQTT clients publishing to a topic may connect to different nodes in the cluster. This means messages for the same topic may be processed by different nodes as a cluster involves multiple nodes processing messages in parallel. Without coordination, it's very challenging to ensure that messages are delivered in the exact order they were published.

• Inconsistent Order During Synchronization:

  In a clustered setup, nodes synchronize data among themselves. Depending on the cluster's synchronization mechanism, the order in which messages are processed or delivered to subscribers may not be strictly maintained. 

• Cluster Load Balancer Behavior:

  The load balancer itself does not maintain state or order of messages; its sole purpose is to distribute connections evenly. It does not consider message ordering.

• Cluster Replication Delays:

  Even if the cluster uses a replication protocol, delays in propagating messages between nodes can cause out-of-order delivery to subscribers.

Achieving Strict Ordering

If strict message ordering is required, consider the following approaches:

• Single Node Responsibility:

  Ensure that all clients publishing and subscribing to a specific topic are routed to the same node. This can be achieved with topic-based routing or a hash-based load-balancing strategy.

• Message Sequencing:

  Include sequence numbers in the message payload. Clients can then reassemble messages in the correct order, even if they arrive out of sequence.

• Use a Dedicated Node for Critical Topics:

  For topics requiring strict ordering, configure the cluster to route all traffic for those topics to a dedicated node.

Eclipse Sparkplug Message Ordering

In an Eclipse Sparkplug ecosystem, Sparkplug Host Applications are responsible for ensuring the correct order of messages received from Edge Nodes. MQTT messages are published to different topics, and depending on the implementation of the MQTT broker, these messages may arrive at the Sparkplug Host Application in a different order than they were originally sent from the Edge Node. This is particularly common when using clustered MQTT brokers like HiveMQ.

It is the responsibility of the Sparkplug Host Application to ensure that all messages are received within a specified Reorder Timeout. In typical environments, this timeout may be as short as a few seconds. However, in deployments with very slow networks or clustered MQTT brokers, a longer timeout may be necessary. In some environments, the MQTT broker may guarantee in-order delivery of QoS0 MQTT messages even across topics, in which case the Reorder Timeout could be set to zero.

Sparkplug uses sequence numbers ("seq")  to track the order of messages. The sequence number is a crucial part of Sparkplug messages. It helps to maintain the order of messages, detect missed messages, and ensure data consistency.

Sparkplug Nodes and Devices are required to maintain this sequence number that increments with each new message (depending on the message type). The sequence numbers are used by Sparkplug Host Applications to ensure the correct sequencing of received data messages from Edge Nodes and Devices. 

For example, if a Sparkplug Host Application receives messages from an Edge Node with sequence numbers 1, 2, and 4, a timer should be started within the Host Application when the message with sequence number 4 arrives. This marks the beginning of the Reordering Timeout period. A message with sequence number 3 must arrive before this timeout expires. If the message with sequence number 3 does not arrive in time, the Host Application should send a Rebirth Request to the Edge Node to reestablish the session state properly. However, if the message with sequence number 3 arrives before the Reordering Timeout expires, the timer can be stopped, and normal operations can resume.

Messages Types That Require a Sequence Number

NBIRTH and NDEATH

The NBIRTH (“Node Birth”) message MUST include a sequence number in the payload and it MUST have a value of 0.

In addition to the “seq” number, a NBIRTH message MUST include a "bdSeq" number as a metric in the payload (“Int64”). This MUST match the “bdSeq” number provided in the MQTT CONNECT packet’s Will Message payload (NDEATH), and the “bdSeq” number MUST start at zero and increment by one on every new MQTT CONNECT packet.

The “bdSeq” number is used to link an NBIRTH message with a DEATH (“Node Death”) message. Since the NDEATH is included in the MQTT CONNECT packet, its timestamp (if present) is not useful for Sparkplug Host Applications. Instead, a “bdSeq” number must be included as a metric in the payload of the NDEATH message. The same “bdSeq” number must also be included in the NBIRTH message published immediately after the MQTT CONNECT.

This ensures that Sparkplug Host Applications can identify which NDEATH message corresponds to a specific NBIRTH message. This correlation is necessary because timing issues with Will Messages may cause NDEATH messages to arrive after a new NBIRTH message. The bdSeq number helps Host Applications determine when an Edge Node should be considered offline.

Note: The NDEATH message does not require an extra sequence number in the payload.

Example:

DBIRTH, NDATA, DDATA, DDEATH 

The DBIRTH (“Device Birth”), NDATA (“Node Data”), DDATA (“Device Data”) and DDEATH (“Device Death”) MUST include a sequence number in the payload. This sequence number MUST have a value of one greater than the previous MQTT message from the Edge Node. This value MUST never exceed 255. If the previous sequence number sent by the Edge Node was 255, the next sequence number sent MUST have a value of 0 and continue to increment.

Trivia: Even if the sequence number specified in the sparkplug.proto schema as “uint64” ("unsigned 64-bit integer”, represents non-negative integer values, ranging from 0 to 18,446,744,073,709,551,615), the Sparkplug 2.0 and 3.0.0 specification requires a 0-255 incremental loop. The reason for this is that no corrections have been made since the first versions of the specification. It has simply been overlooked.

NCMD, DCMD, STATE

NCMD (“Node Command”) and messages are used by Host Applications to write to Edge Node outputs and send Node Control commands to Edge Nodes, and DCMD (“Device Command'') are used to write to device outputs and send Device Control commands to devices. Both message types do not require a sequence number as well as STATE messages. STATE messages published by Sparkplug Host Applications do not even use Sparkplug B payloads and they are used to denote to Edge Nodes whether or not the Sparkplug Host Application is online and operational or not.

HiveMQ Broker Features and Message Ordering

Session Persistence

HiveMQ supports session persistence, which allows the broker to store/queue messages for clients that are not currently connected. When a client reconnects, it receives the stored messages in the order they were received by the broker.

Message Ordering Guarantees

Within a single topic from a single client, HiveMQ maintains the order of messages as they are published and received by the broker. For a client subscribed to a topic, messages are delivered in the same order they were published.

Across multiple topics, MQTT does not provide inherent ordering guarantees since messages from different topics may be processed concurrently.

Client-Level Ordering and Clustered Environment

When multiple clients publish messages to the same topic, the publishing clients can be connected to different HiveMQ cluster nodes. Again, message ordering only applies to a single topic from a single client.

The order of messages with different QoS is uncertain and it is recommended not to rely on any order for messages with different QoS even if the other conditions above are met!

HiveMQ guarantees ordering delivery between QoS 1 and QoS 2 messages, but it does not guarantee ordered delivery between those and parallel QoS 0 messages. Other brokers may implement queues differently.

Session Integrity in a HiveMQ Cluster: If a client with a persistent session using QoS 1 or 2 loses its connection to Node A of the HiveMQ cluster. Upon reconnecting, a load balancer directs the client to Node B. Does HiveMQ guarantee that the client resumes a synchronized session without losing messages and that message ordering is preserved? Yes, this is guaranteed.

HiveMQ is designed for high availability and data integrity, and it ensures a seamless transition in this exact situation through its robust clustering mechanism. All messages that were queued for the client while it was offline are delivered in the correct order, with no data loss.  This architecture provides strong guarantees when a client disconnects and reconnects to a different node.

This even holds true in a session takeover scenario, where a new client connection with the same Client ID forces the disconnection of an older one. The session state remains intact and consistent.

In extreme cases, such as all session replicas being lost, the session is lost as well—but under normal conditions, HiveMQ's replication mechanism ensures message durability and guarantees ordering even across node failures and client reconnections.

Non-Normative:

• If a publisher sends messages in the order of QoS1+2 only like M1 (QoS 1), M2 (QoS 2), M3 (QoS 1), M4 (QoS 2), M5 (QoS 1), a non-shared-subscriber will receive them in exactly the same order like M1, M2, M3,M 4, M5.

• If a publisher sends messages in the order of mixed with QoS 0 like M1 (QoS 1), M2 (QoS 1), M3 (QoS 1), M4 (QoS 0), M5 (QoS 1), a non-shared-subscriber may receive them in this order: M1, M2, M3, M5, M4

• If the connection breaks and messages are re-sent, duplicates are possible, e.g., M1, M2, M3, M2, M3, M4, M5, but the original order is not violated.

Conclusion

Message ordering in MQTT depends on several factors, including QoS level, client connection, and broker configuration, particularly in clustered environments. While HiveMQ preserves and guarantees order within a topic for messages, strict ordering across publishing clients requires additional strategies such as dedicated routing and sequence numbers. Understanding these nuances helps developers and architects design more robust, real-time data pipelines in industrial and IoT systems.