MQTT Essentials Part 3: Client, Broker and Connection Establishment

MQTT Essentials Part 3

Welcome to the thrid edition of MQTT Essentials. A ten-part blog series on the core features and concepts of the MQTT protocol. In this post, we’ll discuss the roles of the MQTT client and broker, and the parameters and options that are available when you connect to a broker.

Last week, we explained how the publish/subscribe pattern works and how it is applied in MQTT. Here’s a quick recap of core concepts:

  • Publish/Subscribe decouples the client that sends a message (the publisher) from the client or clients that receive the messages (the subscribers).

  • MQTT uses the topic (subject) of the message to determine which message goes to which client (subscriber). A topic is a hierarchically-structured string that can be used to filter and route messages (more details).

Our last post gave you a high-level view of the publish/subscribe model and how it differs from a traditional message queue. This post takes a practical approach and is stuffed with basic knowledge about MQTT: definitions for the terms MQTT client and broker, the basics of an MQTT connection, the Connect message with its parameters, and establishing a connection through the acknowledgement of the broker.

Introduction

Because MQTT decouples the publisher from the subscriber, client connections are always handled by a broker. Before we get into the details of these connections, let’s be clear about what we mean by client and broker.

Client

When we talk about a client, we almost always mean an MQTT client. Both publishers and subscribers are MQTT clients. The publisher and subscriber labels refer to whether the client is currently publishing messages or subscribing to messages (publish and subscribe functionality can also be implemented in the same MQTT client). An MQTT client is any device (from a micro controller up to a full-fledged server) that runs an MQTT library and connects to an MQTT broker over a network. For example, the MQTT client can be a very small, resource-constrained device that connects over a wireless network and has a bare-minimum library. The MQTT client can also be a typical computer running a graphical MQTT client for testing purposes. Basically, any device that speaks MQTT over a TCP/IP stack can be called an MQTT client. The client implementation of the MQTT protocol is very straight forward and streamlined. The ease of implementation is one of the reasons why MQTT is ideally suited for small devices. MQTT client libraries are available for a huge variety of programming languages. For example, Android, Arduino, C, C++, C#, Go, iOS, Java, JavaScript, and .NET. You can see a complete list on the MQTT wiki.

Broker

The counterpart of the MQTT client is the MQTT broker. The broker is at the heart of any publish/subscribe protocol. Depending on the implementation, a broker can handle up to thousands of concurrently connected MQTT clients. The broker is responsible for receiving all messages, filtering the messages, determining who is subscribed to each message, and sending the message to these subscribed clients. The broker also holds the sessions of all persisted clients, including subscriptions and missed messages (more details). Another responsibility of the broker is the authentication and authorization of clients. Usually, the broker is extensible, which facilitates custom authentication, authorization, and integration into backend systems. Integration is particularly important because the broker is frequently the component that is directly exposed on the internet, handles a lot of clients, and needs to pass messages to downstream analyzing and processing systems. As discussed in a previous post, subscribing to all message is not really an option. In brief, the broker is the central hub through which every message must pass. Therefore, it is important that your broker is highly scalable, integratable into backend systems, easy to monitor, and (of course) failure-resistant. HiveMQ meets these requirements by using state-of-the-art event-driven network processing, an open plugin system, and standard monitoring providers.

MQTT Connection

The MQTT protocol is based on TCP/IP. Both the client and the broker need to have a TCP/IP stack.

mqtt-tcp-ip-stack

The MQTT connection is always between one client and the broker. Clients never connect to each other directly. To initiate a connection, the client sends a CONNECT message to the broker. The broker responds with a CONNACK message and a status code. Once the connection is established, the broker keeps it open until the client sends a disconnect command or the connection breaks.

MQTT Connection Flow

MQTT connection through a NAT

In many common use cases, the MQTT client is located behind a router that uses network address translation (NAT) to translate from a private network address (like 192.168.x.x, 10.0.x.x) to a public facing address. As we already mentioned, the MQTT client initiates the connection by sending a CONNECT message to the broker. Because the broker has a public address and keeps the connection open to allow bidirectional sending and receiving of messages (after the initial CONNECT), there is no problem at all with clients that are located behind a NAT,

Client initiates connection with the CONNECT message

Now let’s look at the MQTT CONNECT command message. To initiate a connection, the client sends a command message to the broker. If this CONNECT message is malformed (according to the MQTT specification) or too much time passes between opening a network socket and sending the connect message, the broker closes the connection. This behavior deters malicious clients that can slow the broker down.
A good-natured client sends a connect message with the following content (among other things):

MQTT Connect message content

Some information included in a CONNECT message is probably more interesting to implementers of an MQTT library rather than to users of that library. For all the details, have a look at the MQTT 3.1.1 specification.

We will focus on the following options:

ClientId

The client identifier (ClientId) identifies each MQTT client that connects to an MQTT broker. The broker uses the ClientID to identify the client and the current state of the client.Therefore, this ID should be unique per client and broker. In MQTT 3.1.1 (the current standard), you can send an empty ClientId, if you don’t need a state to be held by the broker. The empty ClientID results in a connection without any state. In this case, the clean session flag must be set to true or the broker will reject the connection.

Clean Session

The clean session flag tells the broker whether the client wants to establish a persistent session or not. In a persistent session (CleanSession = false), the broker stores all subscriptions for the client and all missed messages for the client that subscribed with a Quality of Service (QoS) level 1 or 2. If the session is not persistent (CleanSession = true), the broker does not store anything for the client and purges all information from any previous persistent session.

Username/Password

MQTT can send a user name and password for client authentication and authorization. However, if this information isn’t encrypted or hashed (either by implementation or TLS), the password is sent in plain text. We highly recommend the use of user names and passwords together with a secure transport. Brokers like HiveMQ can authenticate clients with an SSL certificate, so no username and password is needed.

Will Message

The last will message is part of the Last Will and Testament (LWT) feature of MQTT. This message notifies other clients when a client disconnects ungracefully. When a client connects, it can provide the broker with a last will in the form of an MQTT message and topic within the CONNECT message. If the client disconnects ungracefully, the broker sends the LWT message on behalf of the client. You can learn more about LWT in part 9 of this series.

KeepAlive

The keep alive is a time interval in seconds that the client specifies and communicates to the broker when the connection established. This interval defines the longest period of time that the broker and client can endure without sending a message.The client commits to sending regular PING Request messages to the broker. The broker responds with a PING response. This method allows both sides to determine if the other one is still available. See part 10 of this series for detailed information about the MQTT keep alive functionality.

That is basically all the information that is neded to connect to an MQTT broker from an MQTT client. Individual libraries often have additional options that can be configured. For example, how queued messages should be stored in a specific implementation.

Broker response with a CONNACK message

When a broker receives a CONNECT message, it is obligated to respond with a CONNACK message.

The CONNACK message contains two data entries:

  • The session present flag

  • A connect acknowledge flag

Session Present flag

The session present flag tells the client whether the broker already has a persistent session available from previous interactions with the client. When a client connects with Clean Session set to true, the session present flag is always false because there is no session available. If a client connects with Clean Session set to false, there are two possibilities: If session information is available for the client Id. and the broker has stored session information, the session present flag is true. Otherwise, if the broker does not have any session information for the client ID, the session present flag is false. This flag was added in MQTT 3.1.1 to help clients determine whether they need to subscribe to topics or if the topics are still stored in a persistent session.

Connect acknowledge flag

The second flag in the CONNACK message is the connect acknowledge flag. This flag contains a return code that tells the client whether the connection attempt was successful..

MQTT Connack contents

Here are the return codes at a glance:.

Return Code Return Code Response
0 Connection accepted
1 Connection refused, unacceptable protocol version
2 Connection refused, identifier rejected
3 Connection refused, server unavailable
4 Connection refused, bad user name or password
5 Connection refused, not authorized

For a more detailed explanation of each of these codes, see the MQTT specification.

Loose ends

You might be wondering how MQTT keeps a connection open even when no messages are being sent, or how you know when a connection is lost. Don’t worry, we devote a whole issue of the essentials series to these topics later on.


That’s the end of part three in our MQTT Essentials series. We hope you learned at least one new thing about MQTT and that you will join us again next week for our post on publishing, subscribing, and unsubscribing in MQTT.

To get the next part of this series as soon as it is released, enter your email in the subscription form below. If you prefer RSS, you can subscribe to our RSS feed here.

52 comments

  1. Ahmed Al-Haddad says:

    Wow you guys. Awesome work seriously!!!

  2. C-: says:

    Can the following sentence be improved?

    “Once the connection is initiated MQTT will keep it open as long as the client doesn’t send a disconnect command or it looses the connection (More details on how this is recognizes is in a later post).”

    C-:

    1. Hi,

      thanks for the feedback! We fixed that sentence and it should be easier to read now.

      Dominik from the HiveMQ Team

  3. Sundeep Saluja says:

    Great Link for a quick go thru the MQTT essential.. Kudos,, Thanks

  4. vj_time says:

    very good work! guys. thank you!

  5. vj_time says:

    great work! Guys! thank you!

  6. sumee says:

    Thanks for the information:-)

  7. vincent says:

    Useful info for new learners.Thanks for the materials.

  8. Nhat Dang says:

    Hi HiveMQ Team, It’s a great topic.
    I’m studying about ESP8266 sending mqtt message to server
    Here is example to send HTTP get message with ESP8266:
    GET /api/xxx HTTP/1.1\r\nHost: xxx.net\r\nContent-Type: application/json\r\nCache-Control: no-cache\r\nConnection: close\r\n\r\n

    so I read the MQTT packet format in http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html#_Toc398718018, I see the packet is built from bytes, I wonder if I could send a MQTT message format like this:
    CONNECT\r\nclientId:”xxx”\r\ncleanSession:true\r\n ….

    1. Hi Nhat Dang,

      this won’t work, but there is the PubSubClient by Nick O’Leary which is a MQTT client that supports the ESP8266.
      You can read a quick introduction here: http://www.hivemq.com/blog/mqtt-client-library-encyclopedia-arduino-pubsubclient/

      Hope that helps,
      Christoph from the HiveMQ Team

    2. Nhat Dang says:

      Hi Christoph ,
      I’m using TM4C123GXL to communicate with esp8266
      and I’m still looking for some library for IDE like ENERGIA or Code Composer Studio (CCS) to program with TM4C
      Do you know any library ???

      Thank you very much

    3. I am not familiar with TM4C.

  9. David says:

    Hiya, thanks for the great post. I am relatively new to MQTT. Actually i’m trying to set a reconnection loop using MQTT and actually i don’t have any clues about how to do it. Example: I’m connected to MQTT broker and sending messages normally, then my connection falls appart, so i lose connection and i need to restart my script again. Is there a way to set a loop so i’ll keep trying to reconnect until my connection is fine again? Thanks in advance!

    1. Hi David,

      Thanks for your comment. Of course that is possible, but that is depending on the MQTT library you are using.

      You can learn more about MQTT libraries and if reconnecting is supported out of the box in our MQTT client encyclopedia (www.hivemq.com/mqtt-client-library-encyclopedia/). If there is no reconnection functionality you need to build it by yourself or contact the developer of the library.

      Hope that helps,
      Christian from the HiveMQ Team

  10. Jason says:

    What is the best design pattern for connecting and sending sensor data in an IoT application with “10s of thousands” of sensors?

    Option:
    a.) Connect and stay connected while continually pushing sensor data to the Broker server
    b.) On a timed basis, say every 30 seconds: connect, push data, disconnect right away

    I would think option a.) would overburden the Broker host with too many connections, while b.) can burden it with continuous re-authentication/authorizations, etc. each time it reconnects. Please advise.

    1. Hi Jason,

      In general I would recommend option a). Keeping connections open is generally quite cheap in comparison to opening new connections (especially if SSL/TLS is in use). And if your sensor data is coming from small restricted devices, which is usually the case, a TLS handshake can be a lot more expensive in terms of CPU, memory and bandwidth than sending a MQTT PING packet every once in a while. Also MQTT brokers are prepared to keep a lot of connections open at the same time.

      Hope that helps,
      Christoph

    2. Stefano says:

      Hi everybody,
      I’ve the same problem: I don’t have any problems of CPU or memory or bandwidth, so I would to implement the ‘b’ pattern (when my sensor produces data, I create a task which connect, push data, disconnect right away and then dies).

      Unfortunately, in this sequence, data never arrive!
      Instead, if I delay the Mqtt disconnect from broker, data arrive with success to broker.

      Am I doing something wrong?
      Thanks in advance.

    3. Hi Stefano,

      it seems like you are closing your client’s connection before the publish is finished.

      You need to know that all MQTT operations are handled asynchronous (Synchronization decoupling). If you want to disconnect right after a publish, you need to implement a callback like “deliveryComplete” on Paho Java client to make sure that the publish is completed.

      Hope this helps,
      Abdullah from the HiveMQ Team.

  11. ravi teja says:

    I’m sending CONNECT to ip 52.29.27.181-> broker.hivemq.com with port 1883 directly through my socket progarm but it is not getting connected .please send the format which i need to send to server through tcp socket.to get CONNACK
    and the data format all commands..

    1. Hallo Ravi,

      an exact definition of the data needed for a connect message can be found in the OASIS MQTT Spec.
      We did experience some difficulties with broker.hivemq.com connectivity in general yesterday, so the error could have been on the broker’s side. These difficulties have been resolved by now and you can connect to it again.

      I hope this helps,
      Florian from the HiveMQ Team

  12. venkat says:

    hai i want to develop a small chat application using mqtt in python programing . how to connect mqtt client using javascript

    1. Hallo venkat,

      We actually have a dedicated blogpost on this matter.

      Good luck with your chat application.

      Kind regards,
      Florian, from The HiveMQ Team.

  13. Issam says:

    Hi,
    How can I change the MQTT broker hostname?

    1. Hallo Isam,

      The hostname is the address used, to reach the machine, the broker is running on.
      So it’s either an IP-Address (for example 127.0.0.1 if you want to connect to a broker, running on your local machine) or a domain that will resolve to the IP-Address. (localhost in the described example)

      Hope that helps.
      Best regards,
      Florian, from The HiveMQ Team.

  14. Amit Kumar says:

    Hi Guys,

    Thanks for the explanation, I want to know how can I run MQTT broker on my own system and accept requests from clients which is nothing but different sensors data?

    Kind Regards
    Amit

    1. Hallo Amit,

      See the installation guide for the HiveMQ broker.
      You can download the broker here.

      Kind regards,
      Florian from The HiveMQ Team.

  15. Maria says:

    Hi,
    I can’t connect to broker.
    the command pronpt keeps showing this message;
    “Connection failed to broker”.
    What’s happening?
    I am new at this, and I’m trying to connect a NodeMCU version 1.0 to the IoT node-red.

    1. Hallo Maria,

      Glad to see you’re taking an interest in MQTT.
      The message you quoted is very generic. My best guess would be that either the broker you connecting to is not running or that you are using a wrong IP-Adress or Port for connection.

      Best regards,
      Florian from the HiveMQ Team.

  16. Sayed says:

    Hi team HiveMQ,
    This is the best mqtt tutorial i have ever found. It clears so many doubts. Still one thing is unclear in my mind is how broker identify the client? For instance i have my esp8266 as a client behind the router and a cluod as mqtt broker. As esp8266 is having a dynamic ip which changes everytime when router get restarted and router’s public ip too. So if i publich a message for this esp8266 how broker will identify if it is the same client? Is there any dependency on client’s ip address? Or is it just the topic on which broker identify the client.

    Thanks.

    1. Hallo Sayed,

      Thank you very much for the kind words. The client is identified by the so called “clientID”, that can be set in the connection options. It is not dependant on the IP-address.

      Kind regards,
      Florian, from The HiveMQ Team

  17. Sherif says:

    Thank you for the post. What happens when two clients with the same clientID connect? My use case is many threads listening to the same topic, do I need a unique ID for each?

    1. Hello Sherif,

      When a second connects, using a clientID that is already in use by a connected client, the ‘old’ client will get disconnected. This mechanism is called “client take over” and is covered in depth on the 10th part of this blog series.
      So yes – unique IDs are a good idea.

      Kind regards,
      Florian from the HiveMQ Team.

  18. Amit Ghodke says:

    how to publish birth message from android mqttclient to kapua server?
    i am able to publish message to kapua server which is using mqtt but on server side my device is not appearing in device list only showing in connections and not message is received at server side following is code .
    can anybody help.

    String topic = “test/”+clientId;
    String payload = “the payload”;
    byte[] encodedPayload = new byte[0];
    try {
    encodedPayload = payload.getBytes(“UTF-8″);
    MqttMessage message = new MqttMessage(encodedPayload);
    message.setRetained(true);

    client.publish(topic, message);
    } catch ( Exception e) {
    e.printStackTrace();
    showToast(clientId+” Error\n”+e.getMessage());
    }

  19. Yogesh Bhatia says:

    Hi,

    I have a question….if i enter a Broker URL, which is not pingable via the MQTT client and MQTT client calls the connection, then my clinet is waiting for 120 seconds before giving error that broker is unreachable. My keepalive time is 60 seconds. Can this 120 seconds time is standard for can i configure it ?

    1. Hi Yogesh,

      The keepAlive value is designed, so the broker can recognise broken connections. When a client does not sent any packets for and and a half times the configured keepAlive value, the broker will disconnect the connection. This value has no impact on the client. The waiting time for a client before it will view a connection as faulty is depending on each individual client implementation.
      There is no MQTT standard time for that.

      Kind regards,
      Florian from The HiveMQ Team.

  20. Yogesh Bhatia says:

    Hi Florian,

    I am presenting my previous question in a different way. If my Mqtt client is trying to connect to a broker url (which doesnt not exist ), mosquitto_connect() is blocked for 120 second (which is too much for my system). Can i somehow reduce that ? Because i cant find any parameter in MQTT api to reduce or remove this blocking call. If i use mosquitto_connect_async(), then i get return value of 0 (which means connection with broker is SUCCESS), even if broker url is invalid.

    1. Hi Yogesh,

      Thank you for clarifying your query.
      Sounds like you are using the mosquitto lib.
      For detailed questions about that library please refer to mosquitto.

      Sorry I can’t be of more help and kind regards,
      Florian from The HiveMQ Team.

  21. Sandhya says:

    Is there a way I can have different authentication implementation for different set of clients? I need one kind of authentication for devices which publish data and another set for the consumers. How do I differentiate between these two?

    1. Hi there,

      nice to see you’re taking an interest in MQTT and HiveMQ.
      There is multiple ways of differentiating between clients. Good practise is to use the clientID for this.

      Kind regards,
      Florian from The HiveMQ Team.

  22. Anuprit says:

    Hi,

    Firstly thanks a lot for this post. I want to set up a local mqtt server and clients connected to it with no internet connection, it will be a local network. Can i use HIVEMQ for this purpose and how will the connection between clients and local server will be a secure connection using HIVEMQ.

    1. Hi Anpurit,

      Thank you for the kind words. It is great to see you are taking an interest in MQTT and HiveMQ.
      You can absolutely use HiveMQ in a local network. Just make sure your to use to HiveMQ’S private IP address, when connecting the clients.
      Regarding security concerns and possibilities I would recommend reading the Security Fundamentals, a blog post series dedicated to the subject.

      Kind regards,
      Florian from The HiveMQ Team.

  23. Biliyaminu Umar says:

    Hello, Good work you are doing. Please how can i use the number of CONNECT and DISCONNECT packets to obtain the number of active, inactive, total and expired clients of the broker. This is assuming i cant get the number of clients directly.

    Thanks

    1. Hi Biliyaminu,

      Nice to see you’re taking an interest in MQTT.

      If you’re using HiveMQ as your MQTT broker than you can use this guide to setup a monitoring dashboard to see all metrics.

      For any other broker solutions we recommend contacting the support of the broker solution your choice.

      Kind regards,
      Abdullah from The HiveMQ Team.

  24. Ziad Kassam says:

    Hi,

    I’m not sure if you have an implementation on MQTT-SN, but if you have, how you can ensure exactly-once delivery over a UDP transport protocol?
    What if an MQTT-SN connection lost, and an ongoing message is on the wire somewhere over the network, a new connection must be created again with a new connection ID, here there is a possibility of message duplication (if the original message received but not acked) as in TCP.

    Regards,
    Ziad

    1. Hi Ziad,

      Nice to see you’re taking an interest in MQTT and HiveMQ.

      The current version of HiveMQ supports only MQTT 3.1 and 3.1.1.

      Kind regards,
      Abdullah from the HiveMQ Team.

  25. Thinker says:

    Hello,

    I want to say a BIG THANK YOU for such a great and systematic written blogs. This is a comprehensive guidance for MQTT and explanation how it is realized and work. Really great appreciation and work!
    I have one question that needs explanation: “How can I sniff the traffic between MQTT Client and Broker when it is encrypted”. For example, I would like to test the connection and publich/receive the data when imported certificates from AWS or with Azure Client, but the question is how can I sniff the traffic between Client and Broker at any time?”

    Many thanks in advance for your answer and support.

    1. Hi Thinker,

      Thank you for the nice words. We appreciate them.

      MQTT doesn’t offer by default any tools to capture traffic. You could however develop a plugin for HiveMQ which captures all the transferred data between the clients and the broker. This solution would also include all encrypted packets.

      Kind regards,
      Abdullah from the HiveMQ Team.

Leave a Reply

Your email address will not be published. Required fields are marked *