MQTT Essentials Part 8: Retained Messages

mqttessentials_part8

Welcome to the eighth part of the MQTT Essentials, a blog series about the core features and concepts in the MQTT protocol. In this post we will introduce retained messages.

When publishing MQTT messages, a publishing client has no guarantee that a message is actually received by a subscribing client. It can only make sure its message gets delivered safely to the broker. The same is true for a subscribing client. If a client is connecting and subscribing to topics it is interested in, there is no guarantee when the subscriber will get the first message, because this totally depends on a publisher on that topic. It can take a few seconds, minutes or hours until the publisher sends a new message on that topic. Until then the subscribing client is totally in the dark about the current status. This is were retained messages come into play.

Retained Messages

A retained message is a normal MQTT message with the retained flag set to true. The broker will store the last retained message and the corresponding QoS for that topic Each client that subscribes to a topic pattern, which matches the topic of the retained message, will receive the message immediately after subscribing. For each topic only one retained message will be stored by the broker.

The subscribing client doesn’t have to match the exact topic, it will also receive a retained message if it subscribes to a topic pattern including wildcards. For example client A publishes a retained message to myhome/livingroom/temperature and client B subscribes to myhome/# later on, client B will receive this retained message directly after subscribing. Also the subscribing client can identify if a received message was a retained message or not, because the broker sends out retained messages with the retained flag still set to true. A client can then decide on how to process the message.

So retained messages can help newly subscribed clients to get a status update immediately after subscribing to a topic and don’t have to wait until a publishing clients send the next update.

In other words a retained message on a topic is the last known good value, because it doesn’t have to be the last value, but it certainly is the last message with the retained flag set to true.

It is important to understand that a retained message has nothing to do with a persistent session of any client, which we covered in the last episode. Once a retained message is stored by the broker, the only way to remove it is explained below.

Send a retained message

Sending a retained message from the perspective of a developer is quite simple and straight-forward. You just need to set the retained flag of a MQTT publish message to true. Each client library typically provides an easy way to do that.

Delete a retained message

There is also a very simple way for deleting a retained message on a topic: Just send a retained message with a zero byte payload on that topic where the previous retained message should be deleted. The broker deletes the retained message and all new subscribers won’t get a retained message for that topic anymore. Often deleting is not necessary, because each new retained message will overwrite the last one.

Why and when you should use Retained Messages ?

A retained message makes sense, when newly connected subscribers should receive messages immediately and shouldn’t have to wait until a publishing client sends the next message. This is extremely helpful when for status updates of components or devices on individual topics. For example the status of device1 is on the topic myhome/devices/device1/status, a new subscriber to the topic will get the status (online/offline) of the device immediately after subscribing when retained messages are used. The same is true for clients, which send data in intervals, temperature, GPS coordinates and other data. Without retained messages new subscribers are kept in the dark between publish intervals. So using retained messages helps to provide the last good value to a connecting client immediately.


So that’s the end of part eight in our MQTT Essentials series. We hope you enjoyed it. In the next post we’ll cover a feature called Last Will and Testament. It makes it possible to send a last message, when a client is disconnected abruptly.

Have a great week and we’ll hope to see you on the next MQTT Monday!

You like the MQTT Essentials? Then sign up for our newsletter and get notified on each new post as soon as its available. If you prefer RSS, you can subscribe to our RSS feed here.

24 comments

  1. Krishanu Banerjee says:

    Hi thanks for the detailed explanation.
    I was curious what happens if publishing client sends a retain message and then a non-retained message.
    as it says “because it doesn’t have to be the last value, but it certainly is the last message with the retained flag set to true”, ?!

  2. Hi Krishanu,

    in that particular case the first retained message will stay retained. The second message will of course get delivered to all subscribers but as soon as new subscribers subscribe to the topic of the messages, the first message will get delivered.

    Hope this clarifies that case,

    Dominik from the HiveMQ Team

  3. sean says:

    Hi, so everytime a new retained message is published, it overwrite the old one right?
    So my temperature sensor should always set the retain flag to true everytime it publish so when someone subscribe, can get the last message. right?

    1. Hi Sean,

      yes, a new retained message on a topic overwrites the old one. In your particular example it makes sense to always send with retained = true.

      All the best,
      Dominik from the HiveMQ Team

  4. Vasudha says:

    Nice blog series :)
    I have a question related to one part of this blog: “For example client A publishes a retained message to myhome/livingroom/temperature and client B subscribes to myhome/# later on, client B will receive this retained message directly after subscribing.”
    Suppose client A publishes a retained message to myhome/livingroom/humidity and myhome/livingroom/power-consumption. Does this mean client B gets retained messages from all three topics?

    1. Hi Vasudha,

      yes this is correct. If the subscriber uses wildcards, multiple retained messages can be received. You can verify this behaviour e.g. by subscribing to root wildcard (“#”) on public brokers like the http://www.mqttdashboard.com. You will receive lots of retained messages then :-)

      All the best,
      Dominik from the HiveMQ Team

  5. Craig says:

    I was curious as to what happens to Retained messages after a broker restart.
    Do Retained messages persist and still get delivered if the broker goes down for any reason and then restarts?
    What would happen with broker restarts for LWT messages?

    Thanks

    1. Hi Craig,

      if you’re using file persistence (which is the default), all retained messages are still available after a broker restart. LWT messages are lost after a broker restart, since the LWT message lifetime ends with the lifetime of a client (and with a restart all clients are disconnected).

      Hope this helps,
      Dominik from the HiveMQ Team

  6. Craig says:

    Hi Dominik,

    Thank you for the response, it did really help.

    I had one other question about Retained messages. Can Retaining too many messages cause performance/memory issues for the broker? For example if you Retained a message for each client that connected and you had a large number of clients using the broker (e.g. tens of thousands, or millions of possible Retained messages) would that start to become problematic for the broker at some point? If so, are there any general guidelines on a rough number of Retained messages a broker should try to stay below to maintain good broker performance?

    Thanks,
    Craig.

    1. Hi Craig,

      this is a very good question! If you don’t use Memory Persistence, even millions of Retained Messages shouldn’t be a problem for the Broker. We recently tested with multiple millions retained messages and HiveMQ works very efficiently without memory overhead. You can have a virtually unlimited amount of retained messages, especially if you’re using the HiveMQ distributed cluster. This may of course not be true for other brokers, though.

      We would recommend to stay below 50-100 retained messages a single client subscription, some MQTT clients may have problems with that. But in general you can use a huge number of retained messages. Do you have a specific use case in mind with that many retained messages? You can contact us via contact@hivemq.com if you want to discuss this in private.

      Thanks,
      Dominik from the HiveMQ Team

  7. ziven says:

    Hi, if client A send many messages to sever, client B is no network, when client B connect network, how to get all messages?
    Thanks

    1. Hi Ziven,

      All messages will be sent to client B automatically after the reconnect if all of the following conditions are met:

      – Client B has been connected before with the clean session flag is false and has subscribe to the topic, where the messages arrive
      – All messages were sent with quality of service greater than 0

      You can read more on how the queuing works here: http://www.hivemq.com/blog/mqtt-essentials-part-7-persistent-session-queuing-messages

      When looking at retained messages only the last message will be persisted on the topic. It will be delivered to a newly subscribing client, right after subscribing.

      Best regards,
      Christian from the HiveMQ Team

  8. Madhu Raj Ojha says:

    Hello HiveMq Team,

    I had a query, I am publishing retained messages on a topic with Qos 0, and then after publishing blank message on the topic with Qos 1 after the job is done so that no stale messages are received by the subscriber after I am done.

    Is it fine to use two different Qos, or should I go with Qos 1 or 0 in both the cases.

    1. Hi Madhu,

      yes, that should work exactly as you described. Bear in mind that Ordered Topic guarantees only apply to QoS 1 and 2 messages, so the QoS 0 message could be sent out of band if QoS 1 messages need a re-send.

      Best,
      Dominik from the HiveMQ Team

  9. Omkar says:

    Is there any way to publish the messages with QoS = 1 using arduino MQTT library?

    1. Hi Omkar,
      unfortunately

      The client only supports publishing at QoS 0:

      as you can see in our Blogpost.

      Kind regards,
      Florian from the HiveMQ Team

  10. Ariane says:

    Hi HiveMQ Team,
    thanks for the great explanation! I have one question I’m confused about: When client A publishes a retained message to for example myhome/livingroom/temperature, it is sent to all subscribers and stored. If after that client A publishes a retained message with that exact content as the last retained message there will be traffic a second time because the second message overwrites the first one and is sent to all subscribers again. Did I get this right or does the broker check if the messages are the same?

    Best regards

    1. Hallo Ariane,

      glad to see you’re taking an interest in MQTT and HiveMQ.
      Your assumption is correct. In the described scenario the retained message will be sent again. According to the MQTTv3.1.1 spec, this is necessary.

      Best regards,
      Florian, from the HiveMQ Team.

  11. chris says:

    If a message is published to a topic with QOS=0 and retain=true. and then later a client subscribes to that that topic with session clean =true. should it receive that message ?

    1. Hallo Chris,

      That is correct. The last message , that has been published on a specific topic, will be sent to any client subscribing on said topic.
      The cleanSession flag does not influence this behaviour, as the retained message is independent from any client sessions.

      Hope that helps.
      Regards,
      Florian, from The HiveMQ Team.

  12. Nikunj says:

    Hi there,
    I have question that suppose there are different messages in message queue and then we send next message with the Qos=0 and retain=False. So Will that message be retained by broker until it publish to any client or removed from broker message queue??

    Thank you.

    1. Hi Nikunj,

      the message will not be queued because of QoS=0
      and the message will not be retained because of retain=false

      Regards,
      The HiveMQ Team

  13. Nick Sebring says:

    Can Anyone help me with the retain flag in Arduino.ide? ESP8266 PubSubClient library

    client.publish(“Charts/PS”, “0”);

    When I use the -r, the sketch does not compile

    It works great when I type in mosquitto_pub -t “topic/subtopic’ -m ‘0’ -t on my PiZeroW command terminal.

Leave a Reply

Your email address will not be published. Required fields are marked *