We’ve been looking into the foundational changes that MQTT 5 introduces and the top reasons MQTT users are upgrading to MQTT 5. We’ve also started to individually examine each of the new features in the MQTT v5 protocol. Let’s continue with another very exciting feature: Shared Subscriptions

At the end of this blog post, we have a video that complements this blog post. We recommend you to read the blog post and watch the video to get a complete understanding of Shared Subscriptions.

MQTT v5 was specifically designed to bridge the gap between what MQTT 3.1.1 offered users and what those users expect from the de facto standard protocol for the Internet of Things. MQTT 5 ensures that the MQTT protocol will keep its place as the standard IoT protocol for decades to come. Similar to Session and Message Expiry Intervals, MQTT users had such high demand for the shared subscription feature that HiveMQ introduced the feature as additional functionality before the MQTT 5 specification was even released. Now, every MQTT broker that wants to conform 100% with the MQTT 5 specification needs to support this feature.

How do Shared Subscriptions work?

Shared subscriptions are an MQTT v5 feature that allows MQTT clients to share the same subscription on the broker. In standard MQTT subscriptions, each subscribing client receives a copy of each message that is sent to that topic. In a shared subscription, all clients that share the same subscription in the same subscription group receive messages in an alternating fashion. This mechanism is sometimes called client load balancing, since the message load of a single topic is distributed across all subscribers.

MQTT clients can subscribe to a shared subscription with standard MQTT mechanisms. All common MQTT clients such as Eclipse Paho can be used without modifying anything on the client side. However, shared subscriptions use a special topic syntax for subscribing.

Shared subscriptions use the following topic structure:

$share/GROUPID/TOPIC

The shared subscription consists of 3 parts:

• A static shared subscription identifier ($share)
• A group identifier
• The actual topic subscriptions (may include wildcards)

A concrete example for such a subscriber would be:
$share/my-shared-subscriber-group/myhome/groundfloor/+/temperature.

Shared Subscriptions In Depth

In shared subscriptions, each subscription group can be conceptually imagined as a virtual client that acts as a proxy for multiple individual subscribers simultaneously. HiveMQ selects one subscriber of the group and delivers the message to this client. By default, a round-robin approach is used. The following picture demonstrates the principle:

In a HiveMQ deployment, there is no limit on the number of shared subscription groups . For example, the following scenario is possible:

In this example there are two different groups with 2 subscribing clients in each shared subscription group. Both groups have the same subscription but have different group identifiers. When a publisher sends a message with a matching topic, one (and only one) client of each group receives the message.

Shared-subscription Use Cases

There are many use cases for shared subscriptions, especially in high-scalability scenarios. The following use case are some of the most popular:

• Client load balancing for MQTT clients that are unable to handle the load on subscribed topics.
• Worker (backend) applications that ingest MQTT streams and need to scale horizontally.
• Reduction of HiveMQ intra-cluster node traffic through optimization of subscriber node-locality for incoming publishes.
• QoS 1 and 2 are used for the delivery semantics but ordered-topic guarantees are not required.
• Hot topics with higher message rates than other topics in the system are causing a scalability bottleneck.

How to subscribe with Shared Subscriptions?

Subscribing clients with shared subscriptions is straight forward. The following command line execution code (that uses the MQTT CLI) shows two MQTT clients that subscribe to the same subscription group and topic:

1
2
3

mqtt sub -h broker.hivemq.com -t '$share/group1/my-share-topic' -i client1 -q 1

mqtt sub -h broker.hivemq.com -t '$share/group1/my-share-topic' -i client2 -q 1

Now, both MQTT clients share a subscription on the topic my-share-topic (they are in the virtual group group1). Each client receives 1/2 of the MQTT messages that are sent over the MQTT broker on the topic my-share-topic.

MQTT clients can join and leave the subscription group at any time. If a third client joins the group, each client receives 1/3 of the relevant MQTT messages.

You can learn more about all the semantics of shared subscriptions in the official HiveMQ Documentation.

Scaling MQTT Subscribers with Shared Subscriptions

The shared subscriptions are a great way to integrate backend systems with plain MQTT (for example, when it’s not feasible to use HiveMQs extension system and dynamic scaling is needed). Shared subscribers can be quickly added whenever they’re needed. An arbitrary number of “worker” applications can be used to consume the messages of the shared-subscription group.

In essence, you get work-distribution in a pushing fashion: the broker pushes the messages to the clients of the shared-subscription group.

Shared Subscriptions are perfect for scaling and load balancing MQTT clients. HiveMQ clusters give you additional advantages in terms of latency and scalability because message routing is optimized internally. Get more information on shared subscriptions and HiveMQ clusters in the official documentation.

Conclusion

We have seen that shared subscriptions are a great way to distribute messages across different MQTT subscribers with standard MQTT mechanisms. This feature lets you add MQTT-client load balancing without any proprietary additions to your MQTT clients. This is especially useful for backend systems or “hot-topics” that can quickly overwhelm a single MQTT client.

Do you already use or plan to use shared subscriptions? Let us know in the comments!

Learn more about MQTT v5

Over the next few weeks, we will continue our technical deep dive into all of the MQTT 5 features. In Part 8 of this series we will be talking about Payload Format Description. If you would like to join us, sign up for our newsletter to get regular updates and ensure that you don’t miss a post.

About Florian Raschbichler

Florian serves as the head of the HiveMQ support team with years of first hand experience overcoming challenges in achieving reliable, scalable, and secure IoT messaging for enterprise customers.
Contact Florian