MQTT Essentials Part 9: Last Will and Testament

mqttessentials_part9

Welcome to the ninth part of MQTT Essentials. A ten-part blog series on the core features and concepts of the MQTT protocol. In this post we explain the Last Will and Testament feature of MQTT.

Because MQTT is often used in scenarios that include unreliable networks, it’s reasonable to assume that some of the MQTT clients in these scenarios will occasionally disconnect ungracefully. An ungraceful disconnect can occur due to loss of connection, empty batteries, or many other reasons. Knowing whether a client disconnected gracefully (with an MQTT DISCONNECT message) or ungracefully (without a disconnect message), helps you respond correctly. The Last Will and Testament feature provides a way for clients to respond to ungraceful disconnects in an appropriate way.

Last Will and Testament

In MQTT, you use the Last Will and Testament (LWT) feature to notify other clients about an ungracefully disconnected client. Each client can specify its last will message when it connects to a broker. The last will message is a normal MQTT message with a topic, retained message flag, QoS, and payload. The broker stores the message until it detects that the client has disconnected ungracefully. In response to the ungraceful disconnect, the broker sends the last-will message to all subscrobed clients of the last-will message topic. If the client disconnects gracefully with a correct DISCONNECT message, the broker discards the stored LWT message.

disconnect

LWT helps you implement various strategies when the connection of a client drops (or at least inform other clients about the offline status).

How do you specify a LWT message for a client?

Clients can specify an LWT message in the CONNECT message that initiates the connection between the client and the broker.

MQTT Connect message content

To learn more about how the connection between the client and broker is established, see part 3 of MQTT Essentials.

When does a broker send the LWT message?

According to the MQTT 3.1.1 specification, the broker must distribute the LWT of a client in the following situations:

  • The broker detects an I/O error or network failure.
  • The client fails to communicate within the defined Keep Alive period.
  • The client does not send a DISCONNECT packet before it closes the network connection.
  • The broker closes the network connection because of a protocol error.

We will hear more about the Keep Alive time in the next post.

Best Practices – When should you use LWT?

LWT is a great way to notify other subscribed clients about the unexpected loss of connection of another client. In real-world scenarios, LWT is often combined with retained messages to store the state of a client on a specific topic. For example, client1 first sends a CONNECT message to the broker with a lastWillMessage that has “Offline” as the payload, the lastWillRetain flag set to true, and the lastWillTopic set to client1/status. Next, the client sends a PUBLISH message with the payload “Online” and the retained flag set to true to the same topic (client1/status). As long as client1 stays connected, newly-subscribed clients to the client1/status topic receive the “Online” retained message. If client1 disconnects unexpectedly, the broker publishes the LWT message with the payload “Offline” as the new retained message. Clients that subscribe to the topic while client1 is offline, receive the LWT retained message (“Offline“) from the broker. This pattern of retained messages keeps other clients up to date on the current status of client1 on a specific topic.


That brings us to the end of part nine in our MQTT Essentials series. Thanks for joining us. In the last post of this series, we’ll cover the MQTT heartbeat mechanism and how the broker knows a client is online or offline.

Have a great week and see you on the next MQTT Monday!

If you enjoy MQTT Essentials sign up for our newsletter and get notified on each new post as soon as its available. If you prefer RSS, you can subscribe to our RSS feed here.

33 comments

  1. Andy says:

    Hi, I am confused that if one client connected to broker I, then this client disconnected from the broker I unexpectedly, at this time this client connected to the broker I again. could the broker send one “last will message”? thanks.

    1. Hi Andy,

      when a client gets disconnected unexpectedly, the LWT will always be sent, except if a client take-over takes place (a client with the same clientId connects and the ‘old’ client gets disconnected).

      If you really want to send a LWT on every disconnect (even graceful disconnects), then you can use a feature of HiveMQ which allows to do this. This blog post explains how to enable this: http://www.hivemq.com/hivemq-2-1-0-hivemq-artifacts-maven-central/

      Hope this helps!
      Dominik from the HiveMQ Team

  2. Sherry says:

    Hi
    I really liked your series regarding MQTT. It provides practical insight to the MQTT protocol and use cases.
    One query regarding the LWT scenario which you have given. It is said that “After the client is connected, the client sets LWT message on the same topic as “offline” and marks this LWT message as retained message “.

    My Query is : If the retained message is changed, then the newly connected clients will still get the retianed message (which is now same as LWTmessage) for when the client is still online. Can you please clairfy this, when the LWT message is made as retained message. Is it practical because of the above scenario.

    Regards
    Sherry

    1. Hi Sherry,

      thanks for the kind words!

      The newly connected clients which subscribe to that topic will receive the retained LWT message, so they can see that the client for that topic is “offline”. If this client comes back again and sets it to “online” again, then all subscribers will receive the message that the clients are online. All new subscribers will also receive the “online” retained message. The broker always sets the LWT message to “offline” if the client disconnects ungracefully. In case the client wants to disconnect gracefully, it needs to set the “offline” message manually (with a separate PUBLISH message), since LWT is only for ungraceful disconnections.

      I hope this clarifies passage.

      Best,
      Dominik from the HiveMQ Team

  3. Per Westermark says:

    I think it would have been better if your description had shown the use of a three-state solution instead of the two-state solution “online” + “offline”.

    A three-state example would be “running”, “stopped”, “dead”.

    On startup, the application registers a last will “dead”.
    It then publishes a persistent state “running”.

    So anyone interested can check the state and see that it is running.

    If the program decides to gracefully end, then it publishes a persistent state “stopped”.

    So anyone interested will know that the program has been run at least once, but currently isn’t available. And the program ended gracefully so there are no technical issues why it isn’t up and running. If the service isn’t available, then it’s probably planned maintainance or similar.

    If the program dies, or something happens with the network, then the broker itself will “replay” the registered last will, changing the persistent state topic to “dead”.

    So anyone interested will know that the program has been run at least once. But something went wrong and the program has died or in other ways failed to communicate. So maybe time to call support and report the failure.
    So anyone interested

    1. Hi Per,

      great example, the three-state solution also works beautifully with retained messages and LWT. Thanks for sharing!

      Best,
      Dominik from the HiveMQ Team

  4. Gerrit says:

    What is the behavior if cluster node has crashed or must be restarted? All client connections to this node are lost. Are LTW messages sent out after restart or from other still active nodes? Otherwise the last retained message on the client status topic is still “online”.

    1. Hi Gerrit,

      this is a great question! The MQTT specification does not cover this case.

      A HiveMQ 3.1 cluster does not send out LWT messages if a node crashes, since the node with the connected clients is responsible for sending out LWTs. The reason for that is, that it’s not possible in distributed systems to detect if a node crashed or if a split-brain scenario occurs. In case of split-brain, sending out LWT messages by other nodes would create wrong state in the cluster in split brain scenarios, because the client is actually connected. In the future there are plans to make this behaviour customizable, since there are some cases where such a split brain behaviour could potentially be beneficial. If a HiveMQ cluster node shuts down gracefully, LWT messages are sent out as expected.

      Best,
      Dominik from the HiveMQ Team

  5. Ronald says:

    Hi,

    Thanks for the nice article, I especially like the ‘online/offline’ best practice, using it now.

    I do not really get the purpose of the last will QoS. The preferred QoS is determined by the subscriber, isn’t it?

    Regards,
    Ronald

    1. Hi Ronald,

      QoS messages are never “upgraded”, so if the original publisher sent a message with QoS 0, a QoS 2 subscriber will still receive the message as QoS 0. You can find more details here: http://www.hivemq.com/blog/mqtt-essentials-part-6-mqtt-quality-of-service-levels

      Hope this helps,
      Dominik

    2. Ronald says:

      Hi Dominik,

      Thanks for your quick response.

      OK, I was not aware that QoS messages are never upgraded. I wonder what the reason for this behavior is. Suppose I have a subscriber for which it is a matter of life and dead that it would receive every message exactly once. The system might even get in trouble if it receives duplicates. In that case it would subscribe at QoS level 2. Is it not strange that the publisher can determine that the QoS level of the communication between the broker and the subscriber should be any lower?

      Given the fact that QoS messages are never upgraded, would it be a best practice to always set the last will QoS to level 2? It will have no effect on the communication between the publisher and the broker, but it will at least not stop the subscribers to reach their preferred QoS level.

      Regards,
      Ronald

    3. Hi Ronald,

      awesome question!

      The main idea is that the initial publisher of a message decides on the importance of a message. If you have a life-or-death scenario, then the publisher should really use QoS 2 (+ timestamps + data integrity checks as discussed in http://www.hivemq.com/blog/mqtt-security-fundamentals-mqtt-message-data-integrity). If you have business logic which needs the message to get upgraded to a different QoS level, you could use the HiveMQ plugin system for doing this.

      Another thing you should bear in mind is, that when the connection does not drop between client and broker, each message, independent of QoS level, will be transmitted exactly once due to TCP guarantees. Only in edge cases (like the client disconnects while messages are transmitted) or you have networks that can ‘blackhole’, the QoS semantics are needed. So please use persistent sessions when you need these guarantees!

      We are strong advocates of QoS 1 instead of QoS 2 whenever possible, but if your scenario absolutely requires QoS 2 then it may be a good idea to always send QoS 2 messages. Unfortunately there is no silver bullet here 🙂

      Hope this helps,
      Dominik from the HiveMQ Team

  6. NHBYK says:

    Dear mqtt users

    If a client subscribed to a topic goes offline, can mqtt send him all data that he missed while he was offline

    best regards

    1. Hi,

      yes this is possible. This blog post discusses all you need to know about it: http://www.hivemq.com/blog/mqtt-essentials-part-7-persistent-session-queuing-messages

      All the best,
      Dominik from the HiveMQ Team

  7. David says:

    Is it possible to update the last will message after a connect, so that state information may be sent that would reflect the state of client at the moment of the ungraceful disconnect.

    connect ( … last_will conatins state init )

    … 10 seconds in the future …

    update LWT

    … network outage …

    updated LWT sent to topics, not the one setup on connect.

    1. Hi David,

      currently there is no consideration of manipulating the LWT messages after the connect message in the mqtt specification.

      Regards,
      Christoph

  8. Ralf says:

    Great article, keep up the awesome work! I do have a question, though: You’re writing

    “[…] the client sets the LWT message on the same topic to the payload “offline” and marks this LWT message as a retained message.”

    I don’t see a retain field in the CONNECT packet. So how does the client set the LWT message to be retained?

    1. Ralf says:

      From how I understand the spec, the retain field of the CONNECT packet is ignored. Can you please clarify!

    2. The spec says:
      “If the Will Flag is set to 1, the Will QoS and Will Retain fields in the Connect Flags will be used by the Server, and the Will Topic and Will Message fields MUST be present in the payload [MQTT-3.1.2-9].”
      “If the Will Flag is set to 0 the Will QoS and Will Retain fields in the Connect Flags MUST be set to zero and the Will Topic and Will Message fields MUST NOT be present in the payload [MQTT-3.1.2-11].”

      This means that the retain field is only ignored if the Will Flag is set to 0.

      Hope that helps,
      Christoph from the HiveMQ Team

    3. Thanks for pointing this out, the retain field is present in the CONNECT packet, but it was missing from our image in the Connect-Part of the series.
      We updated the image to also show the retained field

  9. Gaddour Mohamed says:

    Hi,
    I used mqttws31.js MQTT javascipt client to connect to mosquitto broker with websockets.
    When the client connect to the broker it set WILL message to “offline” and publich message ‘online’ to all subscribers.
    on page onload.
    ths problem is when the page refrech or user click a link in the page and goes to other url. a disconnect event is detected by broker and offline message is send until next page load and then the ‘online’ message is published again.

    So haw to distiguich between Page closed by user (user disconnected) or the user is just navigating from one page to another ?

    1. Hi Mohamed,
      unfortunately when using an Javascript MQTT client in your browser the connection gets lost when loading a new page.
      When a new page is loaded the existing Javascript context gets destroyed and the MQTT client will ungracefully disconnect, sending it’s Last Will message.
      A possible solution to this problem would be to adjust your application in a way so the client gracefully disconnects, when a user navigates to a different page. This way no Last Will message will be sent.

      Kind regards,
      Florian from the HiveMQ Team

  10. Gaddour Mohamed says:

    Hi
    Thanks you for your response, Indeed I can set client.disconnect() on beforeunload js event and The WILL offline message will not be send..
    but haw I can detect that the user has closed the page and it became offline in this case. the status off user will remain always online. I can not find solution for this problem.

    1. Hi Mohamed,
      you will have to specify the event of a user clicking one of your links more detailed, than by the beforeunload event. This way you can have a clean disconnect for users loading a new page, while the Last Will message will be sent by clients, closing the window.

      Hope that is helpful,
      Florian from the HiveMQ Team

  11. Ervin says:

    Hi,
    Thanks for the great site. I am confuse though as to how Retained LWT Messages are updated.
    When a client connects and sets it’s willMessage as online, who’s responsibility is it to update the LWT message e.g. to “offline”? ATM my implementation using M2MQtt client always shows as online.

    1. Hallo Ervin,

      Thank you for the compliment.
      As to your question:

      The LWT message can be specified by each client as part of the CONNECT message, which serves as connection initiation between client and broker.

      In the example mentioned, the LWT is “offline”. Setting that status to “online” can not be done by the LWT, since you can only set the LWT once when connecting.

      I hope that clarifies things.
      Best regards,
      Florian, from The HiveMQ Team.

  12. Lesley Duffey says:

    How does the broker go about forcing the disconnection?

    1. Hi Lesley,
      nice to see you are taking and interest in MQTT and HiveMQ.
      The broker will close the TCP connection.

      Kind regards,
      Florian from The HiveMQ Team.

  13. Charles Lyn says:

    Hi !

    I follow the scenario you explained above for using Last Will and Testament for device status.
    I used the “Offline” text as my last will to the device status topic.

    In on_connect callback, I publish “Online” text to the device status topic.

    My problem is, after the MQTT auto-reconnect (I’m using loop_start() by the way), the “Offline” status from previous last will is still persist. It seems like the on_connect is not called by the auto-reconnect from loop_start().
    While my other topic data are already updated, so I know the device is actually already online.

    I can simply publish the “Online” status periodically. But do you think there is a better way ? Or maybe I have done some mistake in some ways ?

    Thank you.
    Warm regards

    1. Hi Charles,

      Nice to see you are taking an interest in MQTT and HiveMQ.
      Which MQTT Client library and which MQTT broker are you using for your implementation?

      Kind regards,
      Florian, from The HiveMQ Team.

  14. Donatien says:

    Hi,
    Your MQTT Essentials series is awesome. You did a great job explaining simply and clearly how it works.
    I have a question about the ‘lastWillRetain’ flag:
    – in your example’s screenshot, this flag is set to ‘false’
    – in the online / offline example, I understand that this flag has to be true for the broker to publish the LWT message in case of ungraceful disconnection. Is this correct ? Could a client set the lastWillRetain flag to false and this example would still work ?

    In the chronology of the example:
    1. CONNECTING: the client sends a CONNECT message and sets a lastWillMessage of ‘offline’
    2. CONNECTED: the client sends a PUBLISH message with a retained payload ‘online’

    My confusion comes from the fact that I understood a retained message published on a topic will replace last published retained message on the same topic.
    So, here ‘online’ would replace ‘offline’ as retained message for ‘client/status’.
    – if the publisher disconnect ungracefully, if the last retained message is ‘online’, why wouldn’t this be sent by the broker ?
    – Is the lastWillRetained flag a parallel resource, used by the broker in case of ungraceful disconnection ? I don’t clearly grasp the presence of these 2 similar and parallel resources. How are they sed and when by the broker ?

    I hope the question is clearly enough…

    Kind regards,
    Donatien

    1. Hi Donatien,

      Thank you for the kind words. We’re happy to see that you like our MQTT Essentials series.

      You would have right when the retained message of the last will would be set directly after the connection. This problem doesn’t exist though. The last will message is send only after the client has lost his connection. Only then would the retained message replaced by the last will message.

      The chronology would be like this:
      1. The client sends a CONNECT message. Its last will message is stored on the broker (offline).
      2. The client sends a PUBLISH message with a retained payload ‘online’
      3. The client disconnects. The broker replaces the current retained message with the last will message and sends it to the subscriber.

      Hope this clarifies your question.

      Kind regards,
      Abdullah from the HiveMQ Team.

Leave a Reply

Your email address will not be published. Required fields are marked *