MQTT Essentials Part 8: Retained Messages

mqttessentials_part8

Welcome to the eighth part of the MQTT Essentials, a blog series about the core features and concepts in the MQTT protocol. In this post we will introduce retained messages.

When publishing MQTT messages, a publishing client has no guarantee that a message is actually received by a subscribing client. It can only make sure its message gets delivered safely to the broker. The same is true for a subscribing client. If a client is connecting and subscribing to topics it is interested in, there is no guarantee when the subscriber will get the first message, because this totally depends on a publisher on that topic. It can take a few seconds, minutes or hours until the publisher sends a new message on that topic. Until then the subscribing client is totally in the dark about the current status. This is where retained messages come into play.

Retained Messages

A retained message is a normal MQTT message with the retained flag set to true. The broker will store the last retained message and the corresponding QoS for that topic Each client that subscribes to a topic pattern, which matches the topic of the retained message, will receive the message immediately after subscribing. For each topic only one retained message will be stored by the broker.

The subscribing client doesn’t have to match the exact topic, it will also receive a retained message if it subscribes to a topic pattern including wildcards. For example client A publishes a retained message to myhome/livingroom/temperature and client B subscribes to myhome/# later on, client B will receive this retained message directly after subscribing. Also the subscribing client can identify if a received message was a retained message or not, because the broker sends out retained messages with the retained flag still set to true. A client can then decide on how to process the message.

So retained messages can help newly subscribed clients to get a status update immediately after subscribing to a topic and don’t have to wait until a publishing clients send the next update.

In other words a retained message on a topic is the last known good value, because it doesn’t have to be the last value, but it certainly is the last message with the retained flag set to true.

It is important to understand that a retained message has nothing to do with a persistent session of any client, which we covered in the last episode. Once a retained message is stored by the broker, the only way to remove it is explained below.

Send a retained message

Sending a retained message from the perspective of a developer is quite simple and straight-forward. You just need to set the retained flag of a MQTT publish message to true. Each client library typically provides an easy way to do that.

Delete a retained message

There is also a very simple way for deleting a retained message on a topic: Just send a retained message with a zero byte payload on that topic where the previous retained message should be deleted. The broker deletes the retained message and all new subscribers won’t get a retained message for that topic anymore. Often deleting is not necessary, because each new retained message will overwrite the last one.

Why and when you should use Retained Messages ?

A retained message makes sense, when newly connected subscribers should receive messages immediately and shouldn’t have to wait until a publishing client sends the next message. This is extremely helpful when for status updates of components or devices on individual topics. For example the status of device1 is on the topic myhome/devices/device1/status, a new subscriber to the topic will get the status (online/offline) of the device immediately after subscribing when retained messages are used. The same is true for clients, which send data in intervals, temperature, GPS coordinates and other data. Without retained messages new subscribers are kept in the dark between publish intervals. So using retained messages helps to provide the last good value to a connecting client immediately.


So that’s the end of part eight in our MQTT Essentials series. We hope you enjoyed it. In the next post we’ll cover a feature called Last Will and Testament. It makes it possible to send a last message, when a client is disconnected abruptly.

Have a great week and we’ll hope to see you on the next MQTT Monday!

You like the MQTT Essentials? Then sign up for our newsletter and get notified on each new post as soon as its available. If you prefer RSS, you can subscribe to our RSS feed here.

41 comments

  1. Krishanu Banerjee says:

    Hi thanks for the detailed explanation.
    I was curious what happens if publishing client sends a retain message and then a non-retained message.
    as it says “because it doesn’t have to be the last value, but it certainly is the last message with the retained flag set to true”, ?!

  2. Hi Krishanu,

    in that particular case the first retained message will stay retained. The second message will of course get delivered to all subscribers but as soon as new subscribers subscribe to the topic of the messages, the first message will get delivered.

    Hope this clarifies that case,

    Dominik from the HiveMQ Team

    1. Raveendra Seetharam says:

      Just to understand this case, Dominik you are saying the retained message will not be published to the subscriber and only the second message is delivered? This decision is made inside the broker? I have one more question,I have a MQTT demon running and listening. I have a publisher that publishes to a topic and goes back to sleep. I have a subscriber that subscribed to that topic to receive message. I perform the following steps
      1) Run Publisher to publish with retained Flag set to false and goto sleep
      2) Run Subscriber to get message but there is no message delivered by Broker

      The message wasnt retained by deamon. So is retainer flag a mandatory parameter or why didnt broker deliver the message?

    2. Hi Raveendra,

      in short no. Both messages will be delivered to all subscribers. The only difference is that the retained message will stay retained and delivered to all new subscribers.

      In regards to your second question: A publish without the retained flag set to false will only be delivered to his current subscribers. This means that you need to subscribe to your topic first to get the message. If you can’t guarantee that your client will stay connected, you must use cleanSession=false and publish with QoS 1 or 2 to receive all messages that the client missed while it was offline.
      See also MQTT Essentials Part 7 (https://www.hivemq.com/blog/mqtt-essentials-part-7-persistent-session-queuing-messages)

      Hope this helps,
      Abdullah from the HiveMQ Team.

  3. sean says:

    Hi, so everytime a new retained message is published, it overwrite the old one right?
    So my temperature sensor should always set the retain flag to true everytime it publish so when someone subscribe, can get the last message. right?

    1. Hi Sean,

      yes, a new retained message on a topic overwrites the old one. In your particular example it makes sense to always send with retained = true.

      All the best,
      Dominik from the HiveMQ Team

  4. Vasudha says:

    Nice blog series 🙂
    I have a question related to one part of this blog: “For example client A publishes a retained message to myhome/livingroom/temperature and client B subscribes to myhome/# later on, client B will receive this retained message directly after subscribing.”
    Suppose client A publishes a retained message to myhome/livingroom/humidity and myhome/livingroom/power-consumption. Does this mean client B gets retained messages from all three topics?

    1. Hi Vasudha,

      yes this is correct. If the subscriber uses wildcards, multiple retained messages can be received. You can verify this behaviour e.g. by subscribing to root wildcard (“#”) on public brokers like the http://www.mqttdashboard.com. You will receive lots of retained messages then 🙂

      All the best,
      Dominik from the HiveMQ Team

  5. Craig says:

    I was curious as to what happens to Retained messages after a broker restart.
    Do Retained messages persist and still get delivered if the broker goes down for any reason and then restarts?
    What would happen with broker restarts for LWT messages?

    Thanks

    1. Hi Craig,

      if you’re using file persistence (which is the default), all retained messages are still available after a broker restart. LWT messages are lost after a broker restart, since the LWT message lifetime ends with the lifetime of a client (and with a restart all clients are disconnected).

      Hope this helps,
      Dominik from the HiveMQ Team

  6. Craig says:

    Hi Dominik,

    Thank you for the response, it did really help.

    I had one other question about Retained messages. Can Retaining too many messages cause performance/memory issues for the broker? For example if you Retained a message for each client that connected and you had a large number of clients using the broker (e.g. tens of thousands, or millions of possible Retained messages) would that start to become problematic for the broker at some point? If so, are there any general guidelines on a rough number of Retained messages a broker should try to stay below to maintain good broker performance?

    Thanks,
    Craig.

    1. Hi Craig,

      this is a very good question! If you don’t use Memory Persistence, even millions of Retained Messages shouldn’t be a problem for the Broker. We recently tested with multiple millions retained messages and HiveMQ works very efficiently without memory overhead. You can have a virtually unlimited amount of retained messages, especially if you’re using the HiveMQ distributed cluster. This may of course not be true for other brokers, though.

      We would recommend to stay below 50-100 retained messages a single client subscription, some MQTT clients may have problems with that. But in general you can use a huge number of retained messages. Do you have a specific use case in mind with that many retained messages? You can contact us via contact@hivemq.com if you want to discuss this in private.

      Thanks,
      Dominik from the HiveMQ Team

  7. ziven says:

    Hi, if client A send many messages to sever, client B is no network, when client B connect network, how to get all messages?
    Thanks

    1. Hi Ziven,

      All messages will be sent to client B automatically after the reconnect if all of the following conditions are met:

      – Client B has been connected before with the clean session flag is false and has subscribe to the topic, where the messages arrive
      – All messages were sent with quality of service greater than 0

      You can read more on how the queuing works here: http://www.hivemq.com/blog/mqtt-essentials-part-7-persistent-session-queuing-messages

      When looking at retained messages only the last message will be persisted on the topic. It will be delivered to a newly subscribing client, right after subscribing.

      Best regards,
      Christian from the HiveMQ Team

  8. Madhu Raj Ojha says:

    Hello HiveMq Team,

    I had a query, I am publishing retained messages on a topic with Qos 0, and then after publishing blank message on the topic with Qos 1 after the job is done so that no stale messages are received by the subscriber after I am done.

    Is it fine to use two different Qos, or should I go with Qos 1 or 0 in both the cases.

    1. Hi Madhu,

      yes, that should work exactly as you described. Bear in mind that Ordered Topic guarantees only apply to QoS 1 and 2 messages, so the QoS 0 message could be sent out of band if QoS 1 messages need a re-send.

      Best,
      Dominik from the HiveMQ Team

  9. Omkar says:

    Is there any way to publish the messages with QoS = 1 using arduino MQTT library?

    1. Hi Omkar,
      unfortunately

      The client only supports publishing at QoS 0:

      as you can see in our Blogpost.

      Kind regards,
      Florian from the HiveMQ Team

  10. Ariane says:

    Hi HiveMQ Team,
    thanks for the great explanation! I have one question I’m confused about: When client A publishes a retained message to for example myhome/livingroom/temperature, it is sent to all subscribers and stored. If after that client A publishes a retained message with that exact content as the last retained message there will be traffic a second time because the second message overwrites the first one and is sent to all subscribers again. Did I get this right or does the broker check if the messages are the same?

    Best regards

    1. Hallo Ariane,

      glad to see you’re taking an interest in MQTT and HiveMQ.
      Your assumption is correct. In the described scenario the retained message will be sent again. According to the MQTTv3.1.1 spec, this is necessary.

      Best regards,
      Florian, from the HiveMQ Team.

  11. chris says:

    If a message is published to a topic with QOS=0 and retain=true. and then later a client subscribes to that that topic with session clean =true. should it receive that message ?

    1. Hallo Chris,

      That is correct. The last message , that has been published on a specific topic, will be sent to any client subscribing on said topic.
      The cleanSession flag does not influence this behaviour, as the retained message is independent from any client sessions.

      Hope that helps.
      Regards,
      Florian, from The HiveMQ Team.

  12. Nikunj says:

    Hi there,
    I have question that suppose there are different messages in message queue and then we send next message with the Qos=0 and retain=False. So Will that message be retained by broker until it publish to any client or removed from broker message queue??

    Thank you.

    1. Hi Nikunj,

      the message will not be queued because of QoS=0
      and the message will not be retained because of retain=false

      Regards,
      The HiveMQ Team

  13. Nick Sebring says:

    Can Anyone help me with the retain flag in Arduino.ide? ESP8266 PubSubClient library

    client.publish(“Charts/PS”, “0”);

    When I use the -r, the sketch does not compile

    It works great when I type in mosquitto_pub -t “topic/subtopic’ -m ‘0’ -t on my PiZeroW command terminal.

  14. Shahzad says:

    Hi HiveMQ! Thanks for great tutorials.

    I was reading this tutorial and come up a question in my mind. When publisher sends a message to clear a retained message (i.e have to sent an empty payload message with retained flag true) then this empty message will be forwarded to all subscribers or not? I think broker should not forward this empty message as it was just to clear the retain.

    1. Hi Shahzad,

      Glad you like the tutorial.
      Regarding your question: It works the way you think it should work.
      A zero byte payload message will not be forwarded to subscribers. It’s sole purpose is to clear a retained message on a topic.

      Kind regards,
      Florian from The HiveMQ Team.

  15. Shanuka says:

    Hi,
    Thank for the tutorial. I have one question.
    Lets say, If a publisher publish a message with QOS=2 and retained= true.
    While a subscriber already subscribe to the same topic with clear session = false and QOS=2 with the client id=ABC

    So, he will get the message first time. My question if the subscriber unsubscribe and subscribe again with same configuration to the same topic (client id=ABC, clear session = false and QOS=2) will the already published and already delivered retained message during the previous subscription period, be delivered just after the second subscription happens or NOT?

    If NOT, how can I always ensure to get the retained message for every-time I subscribe with the same configuration (client id=ABC, clear session = false and QOS=2).

    1. Hi Shanuka,

      Nice to see you are taking an interest in MQTT and HiveMQ.
      In your described scenario the message will be re-delivered when you subscribe to the topic again.

      Kind regards,
      Florian from The HiveMQ Team.

  16. Jon Greisz says:

    To add to the above. We’re considering MQTT for a battery powered IoT device that communicates once or twice a day. We want to download the configuration whether it has change or not to the device on each communication session, in case for some reason the device lost it’s configuration. Retained messages seem the way to go for this. Would we need to unsubscribe and resubscribe each time, or is there some other method to receive the device configuration each time we wake up and communicate?

    1. Hi Jon,

      this is a good case for retained messages.

      you can also skip the unscubscribe and only do a resubscribe, retained messages are also sent if the subscription already exists.

      excerpt from the MQTT spec:

      If a Server receives a SUBSCRIBE Packet containing a Topic Filter that is identical to an existing Subscription’s Topic Filter then it MUST completely replace that existing Subscription with a new Subscription. The Topic Filter in the new Subscription will be identical to that in the previous Subscription, although its maximum QoS value could be different. Any existing retained messages matching the Topic Filter MUST be re-sent, but the flow of publications MUST NOT be interrupted [MQTT-3.8.4-3].

      Kind regards,
      Christoph from The HiveMQ Team.

    2. Jon Greisz says:

      Thanks much Christoph, that sounds great.

  17. Paolo says:

    Very interesting feature!
    But what happens if a client send a retained message and then later it is powered off.
    How long the retained message is stored in the broker ?
    I mean I don’t want to make a new subscribe and then receive a message, retained, even if the client is off.
    Thanks.

    1. Hi Paolo,

      Nice to see you’re taking an interest in MQTT and HiveMQ.
      A retained message is stored based on a topic. Every time a client subscribes to that topic the retained message will be delivered.
      The retained message will be stored indefinitely for this topic until it is overwritten by a different retained message or it get’s deleted by a null publish.
      Whether or not a client is connected to the broker does not have any impact on the delivery of retained messages.

      I hope this helps. Let me know if anything is still unclear.

      Kind regards,
      Florian from The HiveMQ Team.

  18. Shatavarth Cheruvu says:

    Hi,

    I’m stuck in a little scenario. I have a web app which is subscribed to all the channels on a broker and is meant to listen to every message from the moment it subscribes. Immediately after subscription there’s bombardment of all the retained messages to my listener which I want to discard and consider the messages the listener receives after I complete discarding every previous message.

    Is there any way the broker is informed when to send retained messages selectively or not? Or it by default sends the retained message to every subscriber. Or is there a better solution?

    1. Hi Shatavarth,

      Nice to see you’re taking an interest in MQTT and HiveMQ.
      There is no way to let the broker now not to send retained messages for a specific client or subscriptions.
      In case you want to ignore retained messages, you can check the messages when they are received at the subscriber and ignored them based on the retained flag.

      I hope this helps.
      Kind regards,
      Florian from the HiveMQ Team

  19. Bibhu says:

    Hi,

    Creating an IOT application with MQTT. This application has to show status of all “THINGS” i.e offline or online.For example if Machine1 is connected to Broker it should say ONLINE and if it is disconnected it should say OFFLINE. How can i achieve this using retained messages or any other means.

    Regards

    1. Hi,

      you can combine the Last Will and testament feature of MQTT together with the retained flag to achieve this.
      See the MQTT Essentials Part 9: https://www.hivemq.com/blog/mqtt-essentials-part-9-last-will-and-testament

      Regards,
      The HiveMQ Team

  20. Matthias Zumbroich says:

    Hi,
    provided someone published a retained message on topic xyz and furthermore provided a subscriber has received that message für topic xyz. Provided the subscription to that topic ist still active.
    Now someone deletes the retained message for topic xyz publishing a zero-payload for that topic.
    Does the subscriber get any information (how) about the retained message content he received in the first place not being valid anymore? Is there any chance for the subscriber to keep his internal information about topic xyz up to date in the sence of being equivalent to the information it would receive when resubscribing? My guess: No.
    Thanks a lot in advance.

    1. Hi Mathias,

      Nice to see you’re taking an interest in MQTT. The answer to your question in short is: no.
      The only way to “update” the retained message for a client is to re-subscribe to the topic.

      Kind regards,
      Florian from The HiveMQ Team.

Leave a Reply to Omkar Cancel reply

Your email address will not be published. Required fields are marked *