In the MQTT 5 Essentials, we’re exploring the foundational changes that MQTT v5 introduces and looking at why existing MQTT users are excited about the new version. Our goal is to individually examine each of the new features in the MQTT 5 protocol. Let’s continue with two innovative features: Response Topic and Correlation Data.

We recommend you to watch our video about the Request-Response Pattern at the end of this blog post.

As we’ve seen, modern IoT projects are often complex and usually require collaboration across multiple vendors and teams. Since MQTT is the standard protocol for IoT, improved interoperability and transparency between systems was a key user requirement for MQTT version 5.

Today’s features fulfill part of those user requirements with a standardized solution to achieve a request-response pattern with MQTT.

Request-Response Pattern and MQTT

MQTT is based on asynchronous messaging that follows the publish-subscribe paradigm: Senders and receivers are decoupled from one another in synchronicity, time, and space and one-to-many relationships are possible. It’s important to understand that the request-response pattern of MQTT functions solves problems in a different way than synchronous, one-to-one-based protocols like HTTP.
An MQTT response usually doesn’t “answer” a “question” that the request presents. It is possible to implement a use case for MQTT in a way that is blocking and provides one-to-one messaging that responds with specific information based on parameters of the request. However, in most use cases, the request causes a specific action for the receiver and the response contains the result for this action.
Don’t worry! This will get clearer when we look at a concrete example.

Response Topic

A response topic is an optional UTF-8 string that is available in any PUBLISH or CONNECT packet. In a CONNECT packet, the response topic refers to the WILL publish. If the response topic contains a value, the sender automatically identifies the corresponding PUBLISH as a request.
The response topic field represents the topics on which the responses from the receivers of the message are expected. Both the actual topic of the initial PUBLISH (request) and the response topic can have one or more subscribers. It’s good practice for the sender of the original PUBLISH (request) to subscribe on the contained response topic before sending out the request.

Correlation Data

Correlation data is optional binary data that follows the response topic. The sender of the request uses the data for identifying to which specific request a response that is received later relates. Response topics can be used without correlation data.
Using the correlation makes it possible for the original sender of the request to handle asynchronous responses that can possibly be sent from multiple receivers. This data is irrelevant to the MQTT broker and only functions as a means to identify the relationship between sender and receiver.

Response Information

In the spirit of enabling transparent implementation and better standardization, the MQTT 5 specification introduced the Response Information property. A client can request response information from the broker by setting a boolean field in the CONNECT.
When this request response information is set to true, the broker can send an optional UTF-8 String field (response information) in the CONNACK packet to pass information about the response topics that are expected to be used.
With this feature users can globally define a specific part of your topic tree on the broker, which can then be used by all clients that indicate they want to be using the request-response pattern at connection establishment.

End-to-End Acknowledgement

MQTT ensures that the sender and receiver of messages are completely decoupled. The subscriber (receiver) that gets a message from the broker is in a separate message flow from the publisher (sender) that sends the message to the broker.
There are many use cases that require an acknowledgement of message receipt from the intended recipient. A classic example is opening the door of your smart home. Not only does the sender of the “open door” command (usually a mobile app) want to know when and if the message was received. The sender would also like to know the result of the command.

These so-called “business ACKs” are the primary reason MQTT users were keen on having the request-response pattern introduced to the MQTT 5 specification. MQTT users needed the ability to provide end-to-end acknowledgements between the sender and the receiver of an application message.
Similar to other features in the new protocol specification, this pattern was already used by MQTTv3 users. The introduction of response topics, correlation data, and response information as protocol fields allows for significantly more extensible, dynamic, and transparent application development with the request-response pattern.

Source Code Example

Here is a quick source code example that leverages the HiveMQ MQTT Client:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

//Requester subscribes to response topic
Mqtt5SubAck subAck = requester.subscribeWith()
        .topicFilter("job/client1234/result")
        .send();
        
//Requester publishes request 
Mqtt5PublishResult result = requester.publishWith()
       .topic("job")
       .correlationData("1234".getBytes())
       .responseTopic("job/client1234/result")
       .payload(message.getBytes())
       .send();
       
       
//Responder subscribes to request topic
Mqtt5SubAck subAck = responder.subscribeWith()
        .topicFilter("job")
        .send();
         

//Responder sends response after receiving the request
 Mqtt5PublishResult result = responder.publishWith()
         .topic(publish.getResponseTopic().get())
         .payload(msg.getBytes())
         .correlationData(publish.getCorrelationData().get())
         .send();

Please note, this is not a complete and functioning source code excerpt. The purpose of the example is to demonstrate the intended workflow of the request-response pattern in MQTT. You can find a complete example on GitHub. If you have any questions about the implementation, you are always welcome to join the discussion in our HiveMQ Community Forum.

Summary

• The MQTT request-response pattern is not the same as the request-response of synchronous, client-server based protocols like HTTP.
• Requests as well as responses can have more than one or no subscriber in MQTT.
• Correlation data makes sure that the relation between request and response can be upheld properly.
• This mechanism enables extensible, dynamic, and transparent implementation of “business acknowledgement” functionality.

Best practises and gotchas

• The requester should always subscribe to the response topic before sending the request.
• Use unique identifiers in the response topic.
• Make sure that intended responders and requesters have the necessary permissions to publish and subscribe to the response topics.
  • Reserve a specific part of the topic tree for this purpose and use the response information field to pass it along to the clients.

Learn everything you need to know about MQTT v5

Over the next few weeks, we will continue our technical deep dive into all of the MQTT 5 features. In Part 10 of this series we will be talking about Topic Alias. If you would like to join us, sign up for our newsletter to get regular updates and ensure that you don’t miss a post.

About Florian Raschbichler

Florian serves as the head of the HiveMQ support team with years of first hand experience overcoming challenges in achieving reliable, scalable, and secure IoT messaging for enterprise customers.
Contact Florian