MQTT Essentials Part 7: Persistent Session and Queuing Messages

Persistent Session and Queuing Messages

Welcome to the seventh part of the MQTT Essentials, a blog series about the core features and concepts in the MQTT protocol. In this post we talk about persistent sessions and message queueing in MQTT. Although MQTT is not a message queue per se, it is possible to queue messages for clients.

Persistent Session

When a client connects to a MQTT broker, it needs to create subscriptions for all topics that it is interested in in order to receive messages from the broker. On a reconnect these topics are lost and the client needs to subscribe again. This is the normal behavior with no persistent session. But for constrained clients with limited resources it would be a burden to subscribe again each time they lose the connection. So a persistent session saves all information relevant for the client on the broker. The session is identified by the clientId provided by the client on connection establishment (more details).

So what will be stored in the session?

  • Existence of a session, even if there are no subscriptions
  • All subscriptions
  • All messages in a Quality of Service (QoS) 1 or 2 flow, which are not confirmed by the client
  • All new QoS 1 or 2 messages, which the client missed while it was offlne
  • All received QoS 2 messages, which are not yet confirmed to the client

That means even if the client is offline all the above will be stored by the broker and are available right after the client reconnects.

How to start/end a persistent session?

A persistent session can be requested by the client on connection establishment with the broker. The client can control, if the broker stores the session using the cleanSession flag (see MQTT Essentials part 3 for more information on the connection establishment between client and broker). If the clean session is set to true then the client does not have a persistent session and all information are lost when the client disconnects for any reason. When clean session is set to false, a persistent session is created and it will be preserved until the client requests a clean session again. If there is already a session available then it is used and queued messages will be delivered to the client if available.

How does the client know if there is already a session stored?

Since MQTT 3.1.1, the CONNACK message from the broker contains the session present flag, which indicates to the client if there is a session available on the broker. For detailed information on the connection establishment see part 3 of the MQTT Essentials.

Persistent session on the client side

Similar to the broker, each MQTT client must store a persistent session too. So when a client requests the server to hold session data, it also has the responsibility to hold some information by itself:

  • All messages in a QoS 1 or 2 flow, which are not confirmed by the broker
  • All received QoS 2 messages, which are not yet confirmed to the broker

Best practices

When you should use a persistent session and when a clean session?

Persistent Session

  • A client must get all messages from a certain topic, even if it is offline. The broker should queue the messages for the client and deliver them as soon as the client is online again.

  • A client has limited resources and the broker should hold its subscription, so the communication can be restored quickly after it got interrupted.

  • The client should resume all QoS 1 and 2 publish messages after a reconnect.

Clean session

  • A client is not subscribing, but only publishing messages to topics. It doesn’t need any session information to be stored on the broker and publishing messages with QoS 1 and 2 should not be retried.

  • A client should explicitly not get messages for the time it is offline.

How long are messages stored on the broker ?

A often asked question is how long is a session stored on the broker. The easy answer is until the clients comes back online and receives the message. But what happens if a client does not come online for a long time? The constraint for storing messages is often the memory limit of the operating system. There is no standard way on what to do in this scenario. It totally depends on the use case. In HiveMQ we will provide a possibility to manipulate queued message and purge them.


So that’s the end of part seven in our MQTT Essentials series. We hope you enjoyed it. In the next post we’ll cover Retained Messages. If you already tried out MQTT, you surely noticed the retained flag, when sending a message. Next week we’ll cover what Retained Messages are and how they work.

Have a great week and we’ll hope to see you on the next MQTT Monday!

You like the MQTT Essentials? Then sign up for our newsletter and get notified on each new post as soon as its available. If you prefer RSS, you can subscribe to our RSS feed here.

39 comments

  1. Daniel says:

    Thanks for a super-interesting series!

    Does this mean that if connection A uses a persistent session, and subscribes to topic B and C and then disconnects – and messages continue to flow to B and C, will the client upon reconnection receive the messages in the order they were sent regardless of topic or not?

    In short, is queued messages awaiting the client stored on session-level or on topic-level?

    Regards,
    Daniel

    1. Hi Daniel,

      this is a great question!

      The MQTT 3.1.1 specification states that all topics must be ordered topics. (see http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/mqtt-v3.1.1.html#_Toc385349841). That means, the message ordering must be guaranteed for a topic.

      If you want ordered messages per client, you can set “messages.maxInflight” to the value 1 in the configuration.properties file of HiveMQ 2. This guarantees that all messages are sent ordered.

      Hope this helps!

      Dominik from the HiveMQ Team

  2. Sachin S Sashital says:

    Hi,
    Is it possible to access the (pending) persisted messages for disconnected client connection, or even their count etc., via the plugin APIs ? I couldn’t directed find anything from the documentation.
    Thanks
    Sachin

    1. Sachin S Sashital says:

      Also, can we get information via any APIs, about the persisted messages per topic for offline clients ? I am asking because this is required when we write a plugin that will do push notifications for undelivered messages, for a Chat application, for example.

    2. Hi Sachin,

      this is currently not possible but is scheduled for a later feature version this year together with many other plugin API goodies.

      Best,
      Dominik from the HiveMQ Team

  3. triawan says:

    very interested !
    but could you, to show a real implementation in a source code ? like as when use persistent session is true or false ? and what are differences between when use LWT flag ? and how to debug the clientId, and session ? many thanks before

    1. Hi Triawan,

      you can find implementation examples for all popular programming languages in our MQTT Client Library Encyclopedia. You can find it here: http://www.hivemq.com/mqtt-client-library-encyclopedia

      All the best,
      Dominik from the HiveMQ Team

  4. Thorsten says:

    Hi ,
    I just want to say a big Thank You for this blog. This is by far the best introduction to MQTT I have found. It has answered most of my Newbie questions in a short time.
    Regards
    Thorsten

    1. Hi Thorsten,

      thanks so much for the kind words. Glad to hear you enjoyed the series.

      All the best,
      Dominik from the HiveMQ Team

  5. Saiful Azhar says:

    “In HiveMQ we will provide a possibility to manipulate queued message and purge them.”

    May I know through what method? Do you provide plugin of some sort?

    1. Hi Saiful,

      this is not yet implemented in the released HiveMQ versions.
      It is on our roadmap and will be in one of the upcoming versions.

      Stay tuned.

      Christian from the HiveMQ team

  6. Hello,
    my question is: the session is persisted even if a client performs a graceful disconnection before reconnecting again?
    How can I test the working of a persistent session using the test broker broker.hivemq.com with the websocket client or a stand-alone client like MQTT.fx?
    Thanks in advance.

    1. Hi Massimo,

      You can test persistent session easily by following these steps:

      1. Connect your MQTT client (1) with cleanSession = false
      2. Subscribe to a topic with QoS >0, for example “persistentsession/test/topic”
      3. Disconnect your client
      4. Connect with another client (2) and publish a message to “persistentsession/test/topic” with QoS > 0
      5. Disconnect client 2
      6. Reconnect with client 1
      7. You should receive the message you sent with client 2, while client 1 was offline

      I would use MQTT.fx for client 1 and the HiveMQ websocket client for client 2.

      Hope that helps,
      Christian from the HiveMQ team

    2. Massimo Perrone says:

      Hi Christian,
      thanks for the response. Did you mean that any pending message should be received just after the reconnection, without even subscribe again to the same topic?
      Anyway, it doesn’t work.
      I tried with QoS 1, both on the subscriber and the publisher, set cleanSession to false on the subscriber side, and kept the same clientID on the subsequent reconnection.
      When the subscriber reconnects nothing happens, and nothing happens even after reissuing the subscription.
      I repeated the same test but skipping step 5, that is keeping the publisher connected, but still nothing changed
      (as an aside, I noted that the Clean Session checkbox is disabled (and checked) on the websocket client, but this shouldn’t matter as I used this client for the publisher side).
      Actually, the same happens with an open source broker which I installed on my development machine, so maybe I am misunderstanding the protocol…

    3. Hi Massimo,

      yes, you don’t have to subscribe again.
      When having a persistent session the topics of the client get stored.
      It seems to me that this could be a bug in MQTT.fx.

      Here are example commands for mosquitto_sub/_pub that shows the correct behavior:

      1. Subscribe: mosquitto_sub -h “broker.hivemq.com” -c -i “persistentclient123” -q 1 -t “testtopic/persistent”
      2. Disconnect with Ctrl+C
      3. Publish message: mosquitto_pub -t “testtopic/persistent” -m “test123” -h “broker.hivemq.com” -q 1
      4. Subscribe again (same command as in 1): mosquitto_sub -h “broker.hivemq.com” -c -i “persistentclient123” -q 1 -t “testtopic/persistent”

      Hope that helps,
      Christian from the HiveMQ team

    4. Massimo Perrone says:

      Hello Christian,
      actually using mosquitto_pub/mosquitto_sub commands works as expected.
      I’ll try to dissect the network traffic to see where the difference is.

      Many thanks,
      Massimo Perrone

  7. Tony says:

    Hi,
    The client A connects to a broker(clean session = false) and subscribes topic A and topic C.

    And then the broker publish four messages(QoS 1) to the client A in the following ordering:
    M1(from topic B), M2(from topic C), M3(from topic B), M4(from topic C)

    But the client A disconnect without any reply.

    When the client A reconnect to broker(clean session = false), can the broker publish the four
    messages in another ordering? For example:
    M1(from topic B), M3(from topic B), M2(from topic C), M4(from topic C)
    or
    M2(from topic C), M4(from topic C), M1(from topic B), M3(from topic B)

    1. Hi Tony.

      the broker can publish the four messages in another ordering.
      message ordering is guaranteed on topic level with MQTT.

      Hope that helps,
      Christoph from the HiveMQ team

  8. Ben Kinsella says:

    The article states that Clean Sessions are recommended when “A client is not subscribing, but only publishing messages to topics. It doesn’t has any session informations (sic) to be relevant on the broker”.

    I don’t think this is correct.
    Persistent Sessions (Clean Session = False) have benefits for publishing clients as well as subscribing clients. If the client is attempting to publish messages with QoS 1 or 2, but the connection to the broker is lost, then a Persistent Session will ensure that those messages will be stored, and published when the connection is re-established. If the client is using a Clean Session, then some messages will be lost.

    At least, this is true for the Paho C client that we are using.
    Can you confirm if this is true for the protocol in general?

    1. Hi Ben,

      Thanks for your comment. Very good catch :)

      We have updated the post.

      Best regards,
      Christian from the HiveMQ Team

  9. Java Web Developer says:

    Hi,

    Really nice article, just a kind of suggestion, can we have a pictorial representation (flow chart) or a flow diagram of the system (persistent-session-queuing-messages).
    This will help to easily understand the flow. :-)

  10. Praveen says:

    Hello The HiveMQ Team,

    Thanks for such a nice article. This has really helped me a lot.
    I have one question.
    Terminal 1: I am subscribing for a topic using the command mosquitto_sub -h localhost -c -i “persistentclient123” -q 1 -t “testtopic”
    Terminal 2: I am subscribing for the same topic using the command mosquitto_sub -h localhost -q 1 -t “testtopic” (i.e. -c and -i are missing here)
    Terminal 3: Publishing using the command mosquitto_pub -t “testtopic” -m “test123” -h localhost -q 1

    Use Case:-
    1. Terminal 1 and 2 are disconnected with Ctrl+C
    2. Publish the message from Terminal 3 with the above command
    3. Again connect the Terminal 1 and 2 to localhost

    I am seeing the message “”test123” is only getting displayed for Terminal 1 (and not Terminal 2).
    Does it mean that we need to pass the -c and -i options always while subscribing ?

    1. Hi Praveen,

      glad to hear you like the post.
      You are correct in assuming the ‘-c’ and ‘-i’ options are needed when subscribing, if you want to persist the messages.
      ‘-c’ Sets the cleanSession flag to false, which allows persisting messages. So no ‘-c’ means you will not receive messages, that are published while you are offline.
      And mosquitto_sub requires you to specify a clientId with ‘-i’ when using a persistent session.

      Hope that helps,
      Florian from the HiveMQ Team.

  11. Simon says:

    Hi, is it possible to use a temporary session ID from a flask server to authenticate to a MQTT broker?

    1. Hi Simon,

      I am not very familiar with flask, but you should be able to write a HiveMQ plugin which authenticates against your flask server.

      You can read more about writing plugins here: http://www.hivemq.com/docs/plugins/latest/

      Hope this helps,
      The HiveMQ Team

  12. ulisse says:

    Hello The HiveMQ Team,

    First of all, I would say a great thank your for your tutorial

    By the wa,y I have a question about a statement in the Clean session section in the ‘best practices” chapter
    I keep on being confused by the sentence: <>
    Are Qos1 and 2 messages by default to be retried?

    Thank you in advance
    ulix

    1. Hallo ulisse,

      Thank you for the kind words.
      Messages with a QoS-lelvel of higher than 0 are to be resent by the sender when cleanSession=false (What is called persistent session in the blog post) is used. According to the MQTT spec:

      When a Client reconnects with CleanSession set to 0, both the Client and Server MUST re-send any unacknowledged PUBLISH Packets (where QoS > 0) and PUBREL Packets using their original Packet Identifiers [MQTT-4.4.0-1]. This is the only circumstance where a Client or Server is REQUIRED to redeliver messages.

      So if you don’t want retries, it is best to use cleanSession=true.

      I hope this clarifies things for you.
      Kind regards,
      Florian from The HiveMQ Team.

  13. Kresimir Simunic says:

    Hi,

    I have two questions:

    1. What should happen in the following scenario:

    Client connects with QoS 2 and persistence session. In one moment connection with the broker is lost – for example network connection is lost. Client continues to publish messages to the broker. Then network connection is again established and client reconnects to the broker. Will the client automatically resent the messages sent during time there was no connection to the broker ? Or programmer has to implement it himself ?

    2. Is it possible to have automatically reconnect to the broker (in the case network connection is down) ?

    Thanks.

    1. Hallo Kresimir,

      Glad to see you’re taking an interest in MQTT. You are asking the right questions.
      Regarding question 1.

      The MQTT spec states the following:

      When a Client reconnects with CleanSession set to 0, both the Client and Server MUST re-send any unacknowledged PUBLISH Packets (where QoS > 0) and PUBREL Packets using their original Packet Identifiers [MQTT-4.4.0-1]. This is the only circumstance where a Client or Server is REQUIRED to redeliver messages.

      This means that you do not have to implement your described behaviour, when using a client implementation that is conform with the MQTT 3.1.1 specification.

      Regarding question 2:

      There is nothing about automatic reconnection tries in the MQTT spec. An implementation like that is certainly possible and a good idea, when dealing with unstable networks.
      Many MQTT client implementation already have this functionality included.

      I hope this will help. Best of luck with your MQTT endeavour.

      Kind regards,
      Florian from The HiveMQ Team.

  14. Kresimir Simunic says:

    Hello Florian,

    Thank you very much for your answer.

    I am using M2MQTT Client and it does not resent messages after reconnect (or maybe I am using wrong parameters).

    Is this feature called “Offline Message Buffering” ?

    Because in the description of M2MQTT find here it say that it does not support it. On another side it say that is support OASIS 3.1.1. specification.

    Thanks,

    Kresimir.

    1. Hallo Kresimir,

      The quoted part of the specification has nothing to do with Offline Message Buffering.
      You need to use the option: cleanSession=false when first connecting the client.
      When you are using QoS > 0 and cleanSession=false, the client will resend any non-acknowledged messages.

      Kind regards,
      Florian from The HiveMQ Team.

  15. Kresimir Simunic says:

    Hallo Florian,

    Thank you for the answer.

    I am using cleansesstion=false and QoS=2 when connection and when reconnecting but the messages will not be resent. Does maybe keepalive parameter play any role ? I set it to 0.

    What does then Offline Message Buffering really mean ?

    Thanks,

    Kind Regards,

    Kresimir.

    1. Hallo Kresimir,

      What “Offline Message Buffering” means, is that an MQTT application can handle messages, while it has no connection to the broker, thus being offline.
      Messages will be put into an offline queue and the Application will start sending them, once connection is re-established. The difference between this and the MQTT spec guarantees is, that the messages in the offline buffer do not yet have any outgoing message flow. Not even the PUBLISH packet has been sent yet.

      If you sent a message with QoS=2 and cleanSession=false and the PUBLISH was sent the client has to have some way of remembering this message flow. Should a disconnect happen, before the client receives a corresponding PUBREC, it has to resend this PUBLISH as soon as it reconnects. The same is true for message flows, where a PUBREL was sent but no PUBCOMP was received. In this case the PUBREL needs to be resent.
      One more possible obstacle I can think of, is the fact that you need to use the same clientID, you had before the disconnect, when reconnecting.
      Please bare in mind that this is how the MQTT spec says an implementation has to work. I am not an expert on the M2MQTT implementation. For specific questions about that implementation please refer to the M2MQTT homepage

      Kind regards,
      Florian from The HiveMQ Team.

  16. Kresimir Simunic says:

    Hi Florian,

    Thank you very much for clarifying the issue.

    Does this mean that if the Client lost connection to the Broker and then Client tries to send message to the Broker, PUBLISH will not be send (at least attempted) ?

    And in which moment will Client realize that the network connection was lost ?

    Thanks,

    Best Regards,

    Kresimir.

    1. Hallo Kresimir,

      After the client realised the connection was lost, it will not send a PUBLISH any more. (At least that’s what I believe, from what I read about M2MQTT)
      How and when the client see itself as ‘offline’ is part of the client’s tcp implementation. I can not make any statement about that. Another option would be, that no response was sent to a PINGREQ by the client, which is a non issue for you, when using keepAlive=0

      Kind regards,
      Florian from The HiveMQ Team.

  17. Kresimir Simunic says:

    Hallo Florian,

    Ok, I understand now.

    But then, if client has no connection to the broker and he is trying to publish something – should he not get at least some feedback that the message was not sent – in form of exception or the error code ?

    Kind regards,

    Kresimir Simunic.

    1. Hallo Kresimir,

      Please let me try to illustrate one more time. Let’s assume your described scenario.
      * Client sends PUBLISH (msgID=1) to broker
      * Sometime during this process connection gets lost
      * Since the connection is now lost, client is not able to receive PUBREC (msgID=1)
      * Connection is re-established
      * Client notices, that for the message with msgID=1 it has sent a PUBLISH but no PUBREC was received
      * Client resends PUBLISH (msgID=1) to broker

      To summarise: The lack of a PUBREC (or PUBACK for QoS=1) works as an ‘error code’. The transmission has not been acknowledged and therefore failed.
      I hope this clarifies things.

      Kind regards,
      Florian from The HiveMQ Team.

Leave a Reply

Your email address will not be published. Required fields are marked *