MQTT Essentials Part 3: Client, Broker and Connection Establishment

MQTT Essentials Part 3

Welcome to the third part of the MQTT Essentials. A blog series about the core features and concepts in the MQTT protocol. In this post, we’ll discuss the role of MQTT client and broker and the parameters and options available, when connecting to a broker.

In the last post, we explained how the publish/subscribe pattern works and how it is applied in MQTT. The following is a quick recap of the essence: Publish/Subscribe decouples a client, which is sending a particular message (called publisher) from another client (or more clients), which is receiving the message (called subscriber). In order to determine, which message gets to which client, MQTT uses topics. A topic is a hierarchical structured string, which is used for message filtering and routing (More details).

The last post was really more academical nature as we examined what publish/subscribe is about and how it can be differentiated from a message queuing approach. This post will be way more practical and stuffed with basic knowledge about MQTT. Some topics we discuss are definition of MQTT client & broker, basics of an MQTT connection, the Connect Message with its parameters and the establishing of the connection by the acknowledgement of the broker.

Introduction

As we have seen MQTT decouples publisher and subscriber, so a connection of any client is always with the broker. Before we start diving into the connection details, let’s make clear what we mean by client and broker.

Client

When talking about a client it almost always means an MQTT client. This includes publisher or subscribers, both of them label an MQTT client that is only doing publishing or subscribing. (In general a MQTT client can be both a publisher & subscriber at the same time). A MQTT client is any device from a micro controller up to a full fledged server, that has a MQTT library running and is connecting to an MQTT broker over any kind of network. This could be a really small and resource constrained device, that is connected over a wireless network and has a library strapped to the minimum or a typical computer running a graphical MQTT client for testing purposes, basically any device that has a TCP/IP stack and speaks MQTT over it. The client implementation of the MQTT protocol is very straight-forward and really reduced to the essence. That’s one aspect, why MQTT is ideally suitable for small devices. MQTT client libraries are available for a huge variety of programming languages, for example Android, Arduino, C, C++, C#, Go, iOS, Java, JavaScript, .NET. A complete list can be found on the MQTT wiki.

Broker

The counterpart to a MQTT client is the MQTT broker, which is the heart of any publish/subscribe protocol. Depending on the concrete implementation, a broker can handle up to thousands of concurrently connected MQTT clients. The broker is primarily responsible for receiving all messages, filtering them, decide who is interested in it and then sending the message to all subscribed clients. It also holds the session of all persisted clients including subscriptions and missed messages (More details). Another responsibility of the broker is the authentication and authorization of clients. And at most of the times a broker is also extensible, which allows to easily integrate custom authentication, authorization and integration into backend systems. Especially the integration is an important aspect, because often the broker is the component, which is directly exposed on the internet and handles a lot of clients and then passes messages along to downstream analyzing and processing systems. As we described in one of our early blog post subscribing to all message is not really an option. All in all the broker is the central hub, which every message needs to pass. Therefore it is important, that it is highly scalable, integratable into backend systems, easy to monitor and of course failure-resistant. For example HiveMQ solves this challenges by using state-of-the-art event driven network processing, an open plugin system and standard providers for monitoring.

MQTT Connection

The MQTT protocol is based on top of TCP/IP and both client and broker need to have a TCP/IP stack.

mqtt-tcp-ip-stack

The MQTT connection itself is always between one client and the broker, no client is connected to another client directly. The connection is initiated through a client sending a CONNECT message to the broker. The broker response with a CONNACK and a status code. Once the connection is established, the broker will keep it open as long as the client doesn’t send a disconnect command or it looses the connection.

MQTT Connection Flow

MQTT connection through a NAT

It is a common use case that MQTT clients are behind routers, which are using network address translation (NAT) in order to translate from a private network address (like 192.168.x.x, 10.0.x.x) to a public facing one. As already mentioned the MQTT client is doing the first step by sending a CONNECT message. So there is no problem at all with clients behind a NAT, because the broker has a public address and the connection will be kept open to allow sending and receiving message bidirectional after the initial CONNECT.

Client initiates connection with the CONNECT message

So let’s look at the MQTT CONNECT command message. As already mentioned this is sent from the client to the broker to initiate a connection. If the CONNECT message is malformed (according to the MQTT spec) or it takes too long from opening a network socket to sending it, the broker will close the connection. This is a reasonable behavior to avoid that malicious clients can slow down the broker.
A good-natured client will send a connect message with the following content among other things:

MQTT Connect message content

Additionally there are other informations included in a CONNECT message, which are more a concern to the implementer of a MQTT library than to the user of a library. If you are interested in the details have a look at the MQTT 3.1.1 specification.
So let’s go through all these options one by one:

ClientId

The client identifier (short ClientId) is an identifier of each MQTT client connecting to a MQTT broker. As the word identifier already suggests, it should be unique per broker. The broker uses it for identifying the client and the current state of the client. If you don’t need a state to be hold by the broker, in MQTT 3.1.1 (current standard) it is also possible to send an empty ClientId, which results in a connection without any state. A condition is that clean session is true, otherwise the connection will be rejected.

Clean Session

The clean session flag indicates the broker, whether the client wants to establish a persistent session or not. A persistent session (CleanSession is false) means, that the broker will store all subscriptions for the client and also all missed messages, when subscribing with Quality of Service (QoS) 1 or 2. If clean session is set to true, the broker won’t store anything for the client and will also purge all information from a previous persistent session.

Username/Password

MQTT allows to send a username and password for authenticating the client and also authorization. However, the password is sent in plaintext, if it isn’t encrypted or hashed by implementation or TLS is used underneath. We highly recommend to use username and password together with a secure transport of it. In brokers like HiveMQ it is also possible to authenticate clients with an SSL certificate, so no username and password is needed.

Will Message

The will message is part of the last will and testament feature of MQTT. It allows to notify other clients, when a client disconnects ungracefully. A connecting client will provide his will in form of an MQTT message and topic in the CONNECT message. If this clients gets disconnected ungracefully, the broker sends this message on behalf of the client. We will talk about this in detail in an individual post.

KeepAlive

The keep alive is a time interval, the clients commits to by sending regular PING Request messages to the broker. The broker response with PING Response and this mechanism will allow both sides to determine if the other one is still alive and reachable. We’ll talk about this in detail in a future post.

That are basically all information that are necessary to connect to a MQTT broker from a MQTT client. Often each individual library will have additional options, which can be configured. They are most likely regarding the specific implementation, for example how should queued message be stored.

Broker responds with the CONNACK message

When a broker obtains a CONNECT message, it is obligated to respond with a CONNACK message. The CONNACK contains only two data entries: session present flag, connect return code.

Session Present flag

The session present flag indicate, whether the broker already has a persistent session of the client from previous interactions. If a client connects and has set CleanSession to true, this flag is always false, because there is no session available. If the client has set CleanSession to false, the flag is depending on, if there are session information available for the ClientId. If stored session information exist, then the flag is true and otherwise it is false. This flag was added newly in MQTT 3.1.1 and helps the client to determine, whether it has to subscribe to topics or if these are still stored in his session.

Connect acknowledge flag

The second flag in the CONNACK is the connect acknowledge flag. It signals the client, if the connection attempt was successful and otherwise what the issue is.

MQTT Connack contents

In the following table you see all return codes at a glance.

Return CodeReturn Code Response
0Connection Accepted
1Connection Refused, unacceptable protocol version
2Connection Refused, identifier rejected
3Connection Refused, Server unavailable
4Connection Refused, bad user name or password
5Connection Refused, not authorized

A more detailed explanation of each of these can be found in the MQTT specification.

Loose ends

You maybe ask, how MQTT keeps the connection open, even when there are no messages send? Or how to know when a connection is lost? You have to be patient, but we will devote a whole blog inside of the essentials series to that topic later on.


So that’s the end of part three in our MQTT Essentials series. We hope you learned at least one new thing about MQTT and looking forward to the next post about how publishing, subscribing and unsubscribing works in MQTT.

If you want to be notified as soon as the next part is released, simply sign up for our newsletter below. This brings you fresh content about MQTT and HiveMQ once a week. If you prefer RSS, you can subscribe to our RSS feed here.

27 comments

  1. Ahmed Al-Haddad says:

    Wow you guys. Awesome work seriously!!!

  2. C-: says:

    Can the following sentence be improved?

    “Once the connection is initiated MQTT will keep it open as long as the client doesn’t send a disconnect command or it looses the connection (More details on how this is recognizes is in a later post).”

    C-:

    1. Hi,

      thanks for the feedback! We fixed that sentence and it should be easier to read now.

      Dominik from the HiveMQ Team

  3. Sundeep Saluja says:

    Great Link for a quick go thru the MQTT essential.. Kudos,, Thanks

  4. vj_time says:

    very good work! guys. thank you!

  5. vj_time says:

    great work! Guys! thank you!

  6. sumee says:

    Thanks for the information:-)

  7. vincent says:

    Useful info for new learners.Thanks for the materials.

  8. Nhat Dang says:

    Hi HiveMQ Team, It’s a great topic.
    I’m studying about ESP8266 sending mqtt message to server
    Here is example to send HTTP get message with ESP8266:
    GET /api/xxx HTTP/1.1\r\nHost: xxx.net\r\nContent-Type: application/json\r\nCache-Control: no-cache\r\nConnection: close\r\n\r\n

    so I read the MQTT packet format in http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html#_Toc398718018, I see the packet is built from bytes, I wonder if I could send a MQTT message format like this:
    CONNECT\r\nclientId:”xxx”\r\ncleanSession:true\r\n ….

    1. Hi Nhat Dang,

      this won’t work, but there is the PubSubClient by Nick O’Leary which is a MQTT client that supports the ESP8266.
      You can read a quick introduction here: http://www.hivemq.com/blog/mqtt-client-library-encyclopedia-arduino-pubsubclient/

      Hope that helps,
      Christoph from the HiveMQ Team

    2. Nhat Dang says:

      Hi Christoph ,
      I’m using TM4C123GXL to communicate with esp8266
      and I’m still looking for some library for IDE like ENERGIA or Code Composer Studio (CCS) to program with TM4C
      Do you know any library ???

      Thank you very much

    3. I am not familiar with TM4C.

  9. David says:

    Hiya, thanks for the great post. I am relatively new to MQTT. Actually i’m trying to set a reconnection loop using MQTT and actually i don’t have any clues about how to do it. Example: I’m connected to MQTT broker and sending messages normally, then my connection falls appart, so i lose connection and i need to restart my script again. Is there a way to set a loop so i’ll keep trying to reconnect until my connection is fine again? Thanks in advance!

    1. Hi David,

      Thanks for your comment. Of course that is possible, but that is depending on the MQTT library you are using.

      You can learn more about MQTT libraries and if reconnecting is supported out of the box in our MQTT client encyclopedia (www.hivemq.com/mqtt-client-library-encyclopedia/). If there is no reconnection functionality you need to build it by yourself or contact the developer of the library.

      Hope that helps,
      Christian from the HiveMQ Team

  10. Jason says:

    What is the best design pattern for connecting and sending sensor data in an IoT application with “10s of thousands” of sensors?

    Option:
    a.) Connect and stay connected while continually pushing sensor data to the Broker server
    b.) On a timed basis, say every 30 seconds: connect, push data, disconnect right away

    I would think option a.) would overburden the Broker host with too many connections, while b.) can burden it with continuous re-authentication/authorizations, etc. each time it reconnects. Please advise.

    1. Hi Jason,

      In general I would recommend option a). Keeping connections open is generally quite cheap in comparison to opening new connections (especially if SSL/TLS is in use). And if your sensor data is coming from small restricted devices, which is usually the case, a TLS handshake can be a lot more expensive in terms of CPU, memory and bandwidth than sending a MQTT PING packet every once in a while. Also MQTT brokers are prepared to keep a lot of connections open at the same time.

      Hope that helps,
      Christoph

  11. ravi teja says:

    I’m sending CONNECT to ip 52.29.27.181-> broker.hivemq.com with port 1883 directly through my socket progarm but it is not getting connected .please send the format which i need to send to server through tcp socket.to get CONNACK
    and the data format all commands..

    1. Hallo Ravi,

      an exact definition of the data needed for a connect message can be found in the OASIS MQTT Spec.
      We did experience some difficulties with broker.hivemq.com connectivity in general yesterday, so the error could have been on the broker’s side. These difficulties have been resolved by now and you can connect to it again.

      I hope this helps,
      Florian from the HiveMQ Team

  12. venkat says:

    hai i want to develop a small chat application using mqtt in python programing . how to connect mqtt client using javascript

    1. Hallo venkat,

      We actually have a dedicated blogpost on this matter.

      Good luck with your chat application.

      Kind regards,
      Florian, from The HiveMQ Team.

  13. Issam says:

    Hi,
    How can I change the MQTT broker hostname?

    1. Hallo Isam,

      The hostname is the address used, to reach the machine, the broker is running on.
      So it’s either an IP-Address (for example 127.0.0.1 if you want to connect to a broker, running on your local machine) or a domain that will resolve to the IP-Address. (localhost in the described example)

      Hope that helps.
      Best regards,
      Florian, from The HiveMQ Team.

Leave a Reply

Your email address will not be published. Required fields are marked *