MQTT Essentials Part 7: Persistent Session and Queuing Messages

Persistent Session and Queuing Messages

Welcome to the seventh part of the MQTT Essentials. A ten-part blog series on the core features and concepts of the MQTT protocol. In this post we talk about persistent sessions and message queueing in MQTT. Although MQTT is not a message queue by definition, it can queue messages for clients. We’ll show you how.

Persistent Session

To receive messages from an MQTT broker, a client connects to the broker and creates subscriptions to the topics in which it is interested. If the connection between the client and broker is interrupted during a non-persistent session, these topics are lost and the client needs to subscribe again on reconnect. Re-subscribing every time the connection is interrupted is a burden for constrained clients with limited resources. To avoid this problem, the client can request a persistent session when it connects to the broker. Persistent sessions save all information that is relevant for the client on the broker. The clientId that the client provides when it establishes connection to the broker identifies the session (more details).

What’s stored in a persistent session?

In a persistent session, the broker stores the following information (even if the client is offline). When the client reconnects the information is available immediately.

  • Existence of a session (even if there are no subscriptions).
  • All the subscriptions of the client.
  • All messages in a Quality of Service (QoS) 1 or 2 flow that the client has not yet confirmed.
  • All new QoS 1 or 2 messages that the client missed while offline.
  • All QoS 2 messages received from the client that are not yet completely acknowledged.

How do you start or end a persistent session?

When the client connects to the broker, it can request a persistent session. The client uses a cleanSession flag to tell the broker what kind of session it needs:

  • When the clean session flag is set to true, the client does not want a persistent session. If the client disconnects for any reason, all information and messages that are queued from a previous persistent session are lost.
  • When the clean session flag is set to false, the broker creates a persistent session for the client. All information and messages are preserved until the next time that the client requests a clean session. If the clean session flag is set to false and the broker already has a session available for the client, it uses the existing session and delivers previously queued messages to the client.

For more information on the connection establishment between client and broker, see part 3 of MQTT Essentials.

How does the client know if a session is already stored?

Since MQTT 3.1.1, the CONNACK message from the broker contains a session present flag. This flag tells the client if a previously established session is still available on the broker. For more information on connection establishment, see part 3 of MQTT Essentials.

Persistent session on the client side

Similar to the broker, each MQTT client must also store a persistent session. When a client requests the server to hold session data, the client is responsible for storing the following information:

  • All messages in a QoS 1 or 2 flow, that are not yet confirmed by the broker.
  • All QoS 2 messages received from the broker that are not yet completely acknowledged.

Best practices

Here are some guidelines that can help you decide when to use a persistent session or a clean session:

Persistent Session

  • The client must get all messages from a certain topic, even if it is offline. You want the broker to queue the messages for the client and deliver them as soon as the client is back online.

  • The client has limited resources. You want the broker to store the subscription information of the client and restore the interrupted communication quickly.

  • The client needs to resume all QoS 1 and 2 publish messages after a reconnect.

Clean session

  • The client needs only to publish messages to topics, the client does not need to subscribe to topics.You don’t want the broker to store session information or retry transmission of QoS 1 and 2 messages.

  • The client does not need to get messages that it misses offline.

How long does the broker store messages?

People often ask how long the broker stores the session. The easy answer is: The broker stores the session until the clients comes back online and receives the message. However, what happens if a client does not come back online for a long time? Usually, the memory limit of the operating system is the primary constraint on message storage. There is no standard answer for this scenario. The right solution depends on your use case. For example, in HiveMQ we provide a possibility to purge queued messages.


That finishes part seven in our MQTT Essentials series. Thanks for joining us. Next week, we’ll cover the concept of retained messages and how they work in MQTT. If you’ve already tried MQTT, you probably noticed the retained flag when you were sending a message. In our next post, you can learn all about them.

Have a great week and see you on the next MQTT Monday!

If you like the MQTT Essentials series, sign up for our newsletter to get notified as soon as a new post is available. If you prefer RSS, subscribe to our RSS feed here.

46 comments

  1. Daniel says:

    Thanks for a super-interesting series!

    Does this mean that if connection A uses a persistent session, and subscribes to topic B and C and then disconnects – and messages continue to flow to B and C, will the client upon reconnection receive the messages in the order they were sent regardless of topic or not?

    In short, is queued messages awaiting the client stored on session-level or on topic-level?

    Regards,
    Daniel

    1. Hi Daniel,

      this is a great question!

      The MQTT 3.1.1 specification states that all topics must be ordered topics. (see http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/mqtt-v3.1.1.html#_Toc385349841). That means, the message ordering must be guaranteed for a topic.

      If you want ordered messages per client, you can set “messages.maxInflight” to the value 1 in the configuration.properties file of HiveMQ 2. This guarantees that all messages are sent ordered.

      Hope this helps!

      Dominik from the HiveMQ Team

  2. Sachin S Sashital says:

    Hi,
    Is it possible to access the (pending) persisted messages for disconnected client connection, or even their count etc., via the plugin APIs ? I couldn’t directed find anything from the documentation.
    Thanks
    Sachin

    1. Sachin S Sashital says:

      Also, can we get information via any APIs, about the persisted messages per topic for offline clients ? I am asking because this is required when we write a plugin that will do push notifications for undelivered messages, for a Chat application, for example.

    2. Hi Sachin,

      this is currently not possible but is scheduled for a later feature version this year together with many other plugin API goodies.

      Best,
      Dominik from the HiveMQ Team

  3. triawan says:

    very interested !
    but could you, to show a real implementation in a source code ? like as when use persistent session is true or false ? and what are differences between when use LWT flag ? and how to debug the clientId, and session ? many thanks before

    1. Hi Triawan,

      you can find implementation examples for all popular programming languages in our MQTT Client Library Encyclopedia. You can find it here: http://www.hivemq.com/mqtt-client-library-encyclopedia

      All the best,
      Dominik from the HiveMQ Team

  4. Thorsten says:

    Hi ,
    I just want to say a big Thank You for this blog. This is by far the best introduction to MQTT I have found. It has answered most of my Newbie questions in a short time.
    Regards
    Thorsten

    1. Hi Thorsten,

      thanks so much for the kind words. Glad to hear you enjoyed the series.

      All the best,
      Dominik from the HiveMQ Team

  5. Saiful Azhar says:

    “In HiveMQ we will provide a possibility to manipulate queued message and purge them.”

    May I know through what method? Do you provide plugin of some sort?

    1. Hi Saiful,

      this is not yet implemented in the released HiveMQ versions.
      It is on our roadmap and will be in one of the upcoming versions.

      Stay tuned.

      Christian from the HiveMQ team

  6. Hello,
    my question is: the session is persisted even if a client performs a graceful disconnection before reconnecting again?
    How can I test the working of a persistent session using the test broker broker.hivemq.com with the websocket client or a stand-alone client like MQTT.fx?
    Thanks in advance.

    1. Hi Massimo,

      You can test persistent session easily by following these steps:

      1. Connect your MQTT client (1) with cleanSession = false
      2. Subscribe to a topic with QoS >0, for example “persistentsession/test/topic”
      3. Disconnect your client
      4. Connect with another client (2) and publish a message to “persistentsession/test/topic” with QoS > 0
      5. Disconnect client 2
      6. Reconnect with client 1
      7. You should receive the message you sent with client 2, while client 1 was offline

      I would use MQTT.fx for client 1 and the HiveMQ websocket client for client 2.

      Hope that helps,
      Christian from the HiveMQ team

    2. Massimo Perrone says:

      Hi Christian,
      thanks for the response. Did you mean that any pending message should be received just after the reconnection, without even subscribe again to the same topic?
      Anyway, it doesn’t work.
      I tried with QoS 1, both on the subscriber and the publisher, set cleanSession to false on the subscriber side, and kept the same clientID on the subsequent reconnection.
      When the subscriber reconnects nothing happens, and nothing happens even after reissuing the subscription.
      I repeated the same test but skipping step 5, that is keeping the publisher connected, but still nothing changed
      (as an aside, I noted that the Clean Session checkbox is disabled (and checked) on the websocket client, but this shouldn’t matter as I used this client for the publisher side).
      Actually, the same happens with an open source broker which I installed on my development machine, so maybe I am misunderstanding the protocol…

    3. Hi Massimo,

      yes, you don’t have to subscribe again.
      When having a persistent session the topics of the client get stored.
      It seems to me that this could be a bug in MQTT.fx.

      Here are example commands for mosquitto_sub/_pub that shows the correct behavior:

      1. Subscribe: mosquitto_sub -h “broker.hivemq.com” -c -i “persistentclient123” -q 1 -t “testtopic/persistent”
      2. Disconnect with Ctrl+C
      3. Publish message: mosquitto_pub -t “testtopic/persistent” -m “test123” -h “broker.hivemq.com” -q 1
      4. Subscribe again (same command as in 1): mosquitto_sub -h “broker.hivemq.com” -c -i “persistentclient123” -q 1 -t “testtopic/persistent”

      Hope that helps,
      Christian from the HiveMQ team

    4. Massimo Perrone says:

      Hello Christian,
      actually using mosquitto_pub/mosquitto_sub commands works as expected.
      I’ll try to dissect the network traffic to see where the difference is.

      Many thanks,
      Massimo Perrone

  7. Tony says:

    Hi,
    The client A connects to a broker(clean session = false) and subscribes topic A and topic C.

    And then the broker publish four messages(QoS 1) to the client A in the following ordering:
    M1(from topic B), M2(from topic C), M3(from topic B), M4(from topic C)

    But the client A disconnect without any reply.

    When the client A reconnect to broker(clean session = false), can the broker publish the four
    messages in another ordering? For example:
    M1(from topic B), M3(from topic B), M2(from topic C), M4(from topic C)
    or
    M2(from topic C), M4(from topic C), M1(from topic B), M3(from topic B)

    1. Hi Tony.

      the broker can publish the four messages in another ordering.
      message ordering is guaranteed on topic level with MQTT.

      Hope that helps,
      Christoph from the HiveMQ team

  8. Ben Kinsella says:

    The article states that Clean Sessions are recommended when “A client is not subscribing, but only publishing messages to topics. It doesn’t has any session informations (sic) to be relevant on the broker”.

    I don’t think this is correct.
    Persistent Sessions (Clean Session = False) have benefits for publishing clients as well as subscribing clients. If the client is attempting to publish messages with QoS 1 or 2, but the connection to the broker is lost, then a Persistent Session will ensure that those messages will be stored, and published when the connection is re-established. If the client is using a Clean Session, then some messages will be lost.

    At least, this is true for the Paho C client that we are using.
    Can you confirm if this is true for the protocol in general?

    1. Hi Ben,

      Thanks for your comment. Very good catch 🙂

      We have updated the post.

      Best regards,
      Christian from the HiveMQ Team

  9. Java Web Developer says:

    Hi,

    Really nice article, just a kind of suggestion, can we have a pictorial representation (flow chart) or a flow diagram of the system (persistent-session-queuing-messages).
    This will help to easily understand the flow. 🙂

  10. Praveen says:

    Hello The HiveMQ Team,

    Thanks for such a nice article. This has really helped me a lot.
    I have one question.
    Terminal 1: I am subscribing for a topic using the command mosquitto_sub -h localhost -c -i “persistentclient123” -q 1 -t “testtopic”
    Terminal 2: I am subscribing for the same topic using the command mosquitto_sub -h localhost -q 1 -t “testtopic” (i.e. -c and -i are missing here)
    Terminal 3: Publishing using the command mosquitto_pub -t “testtopic” -m “test123” -h localhost -q 1

    Use Case:-
    1. Terminal 1 and 2 are disconnected with Ctrl+C
    2. Publish the message from Terminal 3 with the above command
    3. Again connect the Terminal 1 and 2 to localhost

    I am seeing the message “”test123” is only getting displayed for Terminal 1 (and not Terminal 2).
    Does it mean that we need to pass the -c and -i options always while subscribing ?

    1. Hi Praveen,

      glad to hear you like the post.
      You are correct in assuming the ‘-c’ and ‘-i’ options are needed when subscribing, if you want to persist the messages.
      ‘-c’ Sets the cleanSession flag to false, which allows persisting messages. So no ‘-c’ means you will not receive messages, that are published while you are offline.
      And mosquitto_sub requires you to specify a clientId with ‘-i’ when using a persistent session.

      Hope that helps,
      Florian from the HiveMQ Team.

  11. Simon says:

    Hi, is it possible to use a temporary session ID from a flask server to authenticate to a MQTT broker?

    1. Hi Simon,

      I am not very familiar with flask, but you should be able to write a HiveMQ plugin which authenticates against your flask server.

      You can read more about writing plugins here: http://www.hivemq.com/docs/plugins/latest/

      Hope this helps,
      The HiveMQ Team

  12. ulisse says:

    Hello The HiveMQ Team,

    First of all, I would say a great thank your for your tutorial

    By the wa,y I have a question about a statement in the Clean session section in the ‘best practices” chapter
    I keep on being confused by the sentence: <>
    Are Qos1 and 2 messages by default to be retried?

    Thank you in advance
    ulix

    1. Hallo ulisse,

      Thank you for the kind words.
      Messages with a QoS-lelvel of higher than 0 are to be resent by the sender when cleanSession=false (What is called persistent session in the blog post) is used. According to the MQTT spec:

      When a Client reconnects with CleanSession set to 0, both the Client and Server MUST re-send any unacknowledged PUBLISH Packets (where QoS > 0) and PUBREL Packets using their original Packet Identifiers [MQTT-4.4.0-1]. This is the only circumstance where a Client or Server is REQUIRED to redeliver messages.

      So if you don’t want retries, it is best to use cleanSession=true.

      I hope this clarifies things for you.
      Kind regards,
      Florian from The HiveMQ Team.

  13. Kresimir Simunic says:

    Hi,

    I have two questions:

    1. What should happen in the following scenario:

    Client connects with QoS 2 and persistence session. In one moment connection with the broker is lost – for example network connection is lost. Client continues to publish messages to the broker. Then network connection is again established and client reconnects to the broker. Will the client automatically resent the messages sent during time there was no connection to the broker ? Or programmer has to implement it himself ?

    2. Is it possible to have automatically reconnect to the broker (in the case network connection is down) ?

    Thanks.

    1. Hallo Kresimir,

      Glad to see you’re taking an interest in MQTT. You are asking the right questions.
      Regarding question 1.

      The MQTT spec states the following:

      When a Client reconnects with CleanSession set to 0, both the Client and Server MUST re-send any unacknowledged PUBLISH Packets (where QoS > 0) and PUBREL Packets using their original Packet Identifiers [MQTT-4.4.0-1]. This is the only circumstance where a Client or Server is REQUIRED to redeliver messages.

      This means that you do not have to implement your described behaviour, when using a client implementation that is conform with the MQTT 3.1.1 specification.

      Regarding question 2:

      There is nothing about automatic reconnection tries in the MQTT spec. An implementation like that is certainly possible and a good idea, when dealing with unstable networks.
      Many MQTT client implementation already have this functionality included.

      I hope this will help. Best of luck with your MQTT endeavour.

      Kind regards,
      Florian from The HiveMQ Team.

  14. Kresimir Simunic says:

    Hello Florian,

    Thank you very much for your answer.

    I am using M2MQTT Client and it does not resent messages after reconnect (or maybe I am using wrong parameters).

    Is this feature called “Offline Message Buffering” ?

    Because in the description of M2MQTT find here it say that it does not support it. On another side it say that is support OASIS 3.1.1. specification.

    Thanks,

    Kresimir.

    1. Hallo Kresimir,

      The quoted part of the specification has nothing to do with Offline Message Buffering.
      You need to use the option: cleanSession=false when first connecting the client.
      When you are using QoS > 0 and cleanSession=false, the client will resend any non-acknowledged messages.

      Kind regards,
      Florian from The HiveMQ Team.

  15. Kresimir Simunic says:

    Hallo Florian,

    Thank you for the answer.

    I am using cleansesstion=false and QoS=2 when connection and when reconnecting but the messages will not be resent. Does maybe keepalive parameter play any role ? I set it to 0.

    What does then Offline Message Buffering really mean ?

    Thanks,

    Kind Regards,

    Kresimir.

    1. Hallo Kresimir,

      What “Offline Message Buffering” means, is that an MQTT application can handle messages, while it has no connection to the broker, thus being offline.
      Messages will be put into an offline queue and the Application will start sending them, once connection is re-established. The difference between this and the MQTT spec guarantees is, that the messages in the offline buffer do not yet have any outgoing message flow. Not even the PUBLISH packet has been sent yet.

      If you sent a message with QoS=2 and cleanSession=false and the PUBLISH was sent the client has to have some way of remembering this message flow. Should a disconnect happen, before the client receives a corresponding PUBREC, it has to resend this PUBLISH as soon as it reconnects. The same is true for message flows, where a PUBREL was sent but no PUBCOMP was received. In this case the PUBREL needs to be resent.
      One more possible obstacle I can think of, is the fact that you need to use the same clientID, you had before the disconnect, when reconnecting.
      Please bare in mind that this is how the MQTT spec says an implementation has to work. I am not an expert on the M2MQTT implementation. For specific questions about that implementation please refer to the M2MQTT homepage

      Kind regards,
      Florian from The HiveMQ Team.

  16. Kresimir Simunic says:

    Hi Florian,

    Thank you very much for clarifying the issue.

    Does this mean that if the Client lost connection to the Broker and then Client tries to send message to the Broker, PUBLISH will not be send (at least attempted) ?

    And in which moment will Client realize that the network connection was lost ?

    Thanks,

    Best Regards,

    Kresimir.

    1. Hallo Kresimir,

      After the client realised the connection was lost, it will not send a PUBLISH any more. (At least that’s what I believe, from what I read about M2MQTT)
      How and when the client see itself as ‘offline’ is part of the client’s tcp implementation. I can not make any statement about that. Another option would be, that no response was sent to a PINGREQ by the client, which is a non issue for you, when using keepAlive=0

      Kind regards,
      Florian from The HiveMQ Team.

  17. Kresimir Simunic says:

    Hallo Florian,

    Ok, I understand now.

    But then, if client has no connection to the broker and he is trying to publish something – should he not get at least some feedback that the message was not sent – in form of exception or the error code ?

    Kind regards,

    Kresimir Simunic.

    1. Hallo Kresimir,

      Please let me try to illustrate one more time. Let’s assume your described scenario.
      * Client sends PUBLISH (msgID=1) to broker
      * Sometime during this process connection gets lost
      * Since the connection is now lost, client is not able to receive PUBREC (msgID=1)
      * Connection is re-established
      * Client notices, that for the message with msgID=1 it has sent a PUBLISH but no PUBREC was received
      * Client resends PUBLISH (msgID=1) to broker

      To summarise: The lack of a PUBREC (or PUBACK for QoS=1) works as an ‘error code’. The transmission has not been acknowledged and therefore failed.
      I hope this clarifies things.

      Kind regards,
      Florian from The HiveMQ Team.

  18. maxi wu says:

    great stuff, easy to follow, yet comprehensive.
    thank you.

    1. Hi Maxi,

      Thanks a lot for the kind words, we appreciate them.

      Kind regards,
      Florian from The HiveMQ Team.

  19. YDDMAX says:

    1. broker publish a message(qos=1,packId=100)
    2. the network is down and broker do not get the PUBACK.
    3. five days passed
    4. client connect to the broker again,the broker must republish the message(qos=1 packId=100)?

    the packId of the republished message must be 100?

    1. Hi,

      Nice to see you’re taking an interest in MQTT and HiveMQ.

      Your assumption is right. The packId will stay the same.

      When a Client reconnects with CleanSession set to 0, both the Client and Server MUST re-send any unacknowledged PUBLISH Packets (where QoS > 0) and PUBREL Packets using their original Packet Identifiers

      Source: MQTT 3.1.1 Chapter 4.4

      Kind regards,
      Abdullah from the HiveMQ Team.

  20. Naveen Rai says:

    Hi HIVEMQ Team,

    1) What if a client publishes to a topic using Qos 0 but another client has subscribed to the same topic with Qos 1/2, in this case would the subscriber be able to retrieve these messages from the broker upon re-connection (with persistent session true)? i.e. what happens during QoS Upgrading?

    2)If two clients publish to the same topic with same/different QoS level then what will the subscriber , subscrbed to the same topic receive? Data from Pub1 or Pub2?

    Thanks,
    Naveen

    1. Hi Naveen,

      Nice to see you’re taking an interest in MQTT and HiveMQ.

      1. Due to the used publish QoS level of 0 will the messages not be queued for the subscriber and therefore not received after a reconnect. The QoS level can only be downgraded. A upgrade isn’t possible.

      2. The subscriber receives all publishes to the subscribed topic. That includes published with different QoS levels.

      Kind regards,
      Abdullah from the HiveMQ Team.

  21. ReGo says:

    Nice blog!

Leave a Reply

Your email address will not be published. Required fields are marked *