MQTT 101 – How to Get Started with the lightweight IoT Protocol

Introduction

Everybody talks about the Internet of Things nowadays. Increasingly affordable micro controllers like Arduino and Raspberry Pi are enabling cheap devices that measure sensor data and send it over the internet. The goal of this post is to introduce the lightweight protocol MQTT and its capabilities to send data between devices and other systems and to demonstrate them by implementing two clients with Eclipse Paho.

The term Internet of Things was first used by Kevin Ashton in 2009 for interconnecting physical devices over the internet. The basic idea is very simple: Physical devices can exchange data between each other or being controlled by others. Examples of such devices would be a refrigerator, a car, a building or basically any other electronic device. One of the most common use cases is the collection, transmission, consolidation and displaying of sensor data. The results could be a web dashboard with the aggregated values or an alarm, when a threshold is exceeded.
The application scenarios are almost unlimited. Imagine your alarm clock would know that your train to work is 15 minutes late and adjust itself accordingly. Also your coffee maker is switched on automatically 15 minutes later to make you a hot cup of coffee before you leave for work. Sounds like the future ? All that is already possible today. Ericsson predicts that in 2020 50 billion devices are connected over the internet. The communication between the huge amount of devices is enabled by IPv6 and lightweight communication protocols like MQTT.

MQTT

MQTT was developed by Andy Stanford-Clark (IBM) and Arlen Nipper (Eurotech; now Cirrus Link) in 1999 for the monitoring of an oil pipeline through the desert. The goals were to have a protocol, which is bandwidth-efficient and uses little battery power, because the devices were connected via satellite link and this was extremely expensive at that time.
The protocol uses a publish/subscribe architecture in contrast to HTTP with its request/response paradigm. Publish/Subscribe is event-driven and enables messages to be pushed to clients. The central communication point is the MQTT broker, it is in charge of dispatching all messages between the senders and the rightful receivers. Each client that publishes a message to the broker, includes a topic into the message. The topic is the routing information for the broker. Each client that wants to receive messages subscribes to a certain topic and the broker delivers all messages with the matching topic to the client. Therefore the clients don’t have to know each other, they only communicate over the topic. This architecture enables highly scalable solutions without dependencies between the data producers and the data consumers.

MQTT Publish/Subscribe

MQTT Publish/Subscribe Architecture

The difference to HTTP is that a client doesn’t have to pull the information it needs, but the broker pushes the information to the client, in the case there is something new. Therefore each MQTT client has a permanently open TCP connection to the broker. If this connection is interrupted by any circumstances, the MQTT broker can buffer all messages and send them to the client when it is back online.
As mentioned before the central concept in MQTT to dispatch messages are topics. A topic is a simple string that can have more hierarchy levels, which are separated by a slash. A sample topic for sending temperature data of the living room could be house/living-room/temperature. On one hand the client can subscribe to the exact topic or on the other hand use a wildcard. The subscription to house/+/temperature would result in all message send to the previously mention topic house/living-room/temperature as well as any topic with an arbitrary value in the place of living room, for example house/kitchen/temperature. The plus sign is a single level wild card and only allows arbitrary values for one hierarchy. If you need to subscribe to more than one level, for example to the entire subtree, there is also a multilevel wildcard (#). It allows to subscribe to all underlying hierarchy levels. For example house/# is subscribing to all topics beginning with house.

Eclipse Paho

Now with the concept of topics explained, it is time to jump right into the first implementation and to show how the publishing and subscribing can be done using source code. But first a quick introduction of Eclipse Paho, the MQTT implementation used in this example. It has been founded under the umbrella of the Eclipse Foundation at the beginning of 2012 with the goal to provide open IoT protocol implementations. The implementation of MQTT is the de-facto reference implementation and available in Java, C, C++, JavaScript, Lua, Python and soon also C#. The origin of most of the implementations is the codebase of IBM and Eurotech, who have used them in many internal projects in production.

Use Case

In order to make the subsequent code more understandable, we will use the transferring of sensor data from a temperature and brightness sensor to a control center over the internet as an example. The sensors will be connected to a Raspberry Pi, which acts as gateway to the MQTT broker, which resides in the cloud. On the other side is a second device, the control center, that also has an MQTT client and receives the data. Additionally we will implement a notification, which alerts the control center if the sensor is disconnected.

Use Case Communication Flow

Communication between the sensor client and the control center over MQTT

Implementation

First we will implement the sensor client, which is simulating a thermometer and a brightness sensor. It should send a current value every second to the MQTT broker. In this case we are using the HiveMQ public broker on the MQTT Dashboard. The first step is to create an instance of the MqttClient class.

As parameters for the constructor it is necessary to specify the URL of the broker (tcp://broker.mqttdashboard.com:1883) and also the client id. The latter is an unique identifier overall the broker. A good choice for the client id is the MAC address of the computer, because that is automatically unique. In the example -pub is added to the mac address, because otherwise it wouldn’t work starting both clients on the same machine for testing. After creating the instance, it is possible to try to connect to the broker with calling client.connect(). Apart from a simple connect, it is also possible to hand over more parameters. One example is the clean session flag. When it is set to true, the broker will wipe the session every time the client disconnects, otherwise it will keep the subscription and buffer the messages sent with Quality of Service 1 and 2 (more on this later). The session is assigned to the client id, therefore it is truly important to have it unique. Another option is Last Will and Testament (LWT), which helps detecting failures of other clients. As stated above every client has an open connection, so when the client disconnects ungracefully the broker can detect that. If the client has set a LWT topic and message on connect, the broker will send that to the specified topic, which allows the client to notify others about its failure.

Now we have to implement the business logic to retrieve the values and send them every second, therefore we use an infinite loop and the methods publishTemperature and publishBrightness. Each method creates a MqttTopic object and a random value, which will then be published. The complete examples of both clients can be found on GitHub.

Each message can be published with one of three quality of service levels (QoS). These levels are associated with different guarantees. A message send with level 0 doesn’t have a guarantee at all, it implies fire and forget. Level 1 guarantees that the message will at least arrive once, but can arrive more than once. Level 2 is the most sophisticated choice, which guarantees that the message arrives at the destination exactly once. The choice of QoS is a trade-off, between protocol overhead and the guarantee that the message arrives, because ensuring QoS 2 is using more bandwidth than QoS 0.

The next step is implementing the subscribing client, which is reading the values on the topics home/temperature and “home/brightness”, plus observes the home/LWT topic for the last will message to detect a failure of the sensor client. The initialization of the MqttClient instance is almost the same, except we use -sub as a suffix for the client id. For receiving the messages sent by the sensor simulator, it is necessary to implement the MqttCallback interface. The MqttCallback interface defines three methods that need to implemented: connectionLost, messageArrived and deliveryComplete. connectionLost is called when the connection is unexpectedly closed form the MQTT broker. This method is the best place for a reconnect logic. The method messageArrived is called when the broker sends a new message to this particular client. Finally, there is the deliveryComplete method that is called after a message with QoS 1 or 2 reaches the broker. For our use case implementing the messageArrived method is enough. All arriving messages should be print out with topic and payload and if it is sent on the home/LWT topic, we additionally print out Sensor gone!.

After implementing the callback, we have to make it known to the MqttClient before connecting. Also after the successful connection is established, it is necessary to subscribe to all topics, which should be sent to the client. In this case it should be everything that starts with home, so the usage of a multi-level wildcard saves us from subscribing to 3 different topics: home/#.

Run both clients

Now that the sensor client and the Control Center are implemented, it is time to run both applications. We start the Control Center first and then the sensor client. Once the sensor client starts, messages arrive at the Control Center. If you will now exit the sensor client ungracefully, this is recognized by the Control Center immediately through the Last Will and Testament message.
Congratulations, you have now build your first Internet of Things application with MQTT!

Summary

This simple application is just an example and can be easily adapted and reused for more advanced purposes. You could simply extend the existing code so that multiple clients in different rooms measure the temperature and brightness, and the Control Center also indicates that. Another possibility would be to replace the Control Center through a web dashboard where the values are displayed in several charts using the Paho JavaScript library.

As we have seen, it is possible with very little effort to create application that communicates over MQTT. The Eclipse Paho library implements all the functionality that is described in the MQTT specification and MQTT should be a tool in the repertoire of every developer, who deals with the Internet of Things or the requirement of connecting mobile devices.

54 comments

  1. Nitin says:

    Nice article! :)

  2. atze says:

    This is the first article about mqtt that I fully understand.
    Most articles start of by stating how easy it is and than engulf me with loads of incomprehensible programming.

    Too bad it isn’t in python.

    Well done.

  3. Giovanni Quarella says:

    Very good article.

  4. Srinivasa Raju V says:

    Very nicely narrated and easy to adopt. Thanks and provide more posts like this

  5. Simz says:

    Hi, I’m using eclipse IDE. Can you please help using running this Mqtt in this IDE? I’m getting compilation errors on this when I try to use source code in eclipse.

    1. Hi Simz,

      what exact error do you get? Did you include the Eclipse Paho library?

      Dominik from the HiveMQ Team

  6. amit deshmukh says:

    GOOD ARTICLE.

  7. Karthikeyan says:

    Hi,
    Does HiveMQ support QoS1 messages?

    Regards,
    Karthik.

    1. Hi Karthik,

      yes, HiveMQ does support QoS 1. In fact, HiveMQ supports the full MQTT specification and is 100% compliant. That means, that QoS 2 is of course also supported.

      Best,
      Dominik from the HiveMQ Team

  8. Aqib says:

    Very nice and comprehensive one! Just started to reading about MQTT today and found this article. Gave a boost :)

  9. Aqib says:

    What about C# .net?

    1. Hi,

      Paolo Patierno, the developer of the popular M2MQTT library (now under the Paho umbrella) has written a fantastic blog post about the usage with C# here: http://www.hivemq.com/blog/mqtt-client-library-encyclopedia-m2mqtt

      Hope this helps!
      Dominik from the HiveMQ team

  10. venkatesh says:

    Hi,

    i’m a and engineering student. i have seen all the blog post of mqtt essentials. They are really awesome. with that mqqt essentials blog posts, i got to know about mqtt terminology and basic understanding about mqtt.

    My Issue:
    i have raspberry pi. i want to connect temperature sensor to it. please guide me the source code to be involved in writing in general python code. because we generally write a rpi code in python no.i said i’m student.
    1. please tell me how to write a such a python code publishing temperature to the HiveMQ mqtt broker?
    and also on other side
    2.how to subscribe for that temperature readings through my mobile or other alternative?

  11. Hi venkatesh,

    Roger Light, author of the library has a good blog post on Paho Python here: http://www.hivemq.com/blog/mqtt-client-library-paho-python

    Dominik from the HiveMQ Team

  12. Deepak says:

    Really usefull !Can you please tell me how to set up the broker for windows?

    1. Hi Deepak,

      thanks for the feedback! You can see the installation instructions here: http://www.hivemq.com/docs/hivemq/latest/#hivemqdocs_installation_for_specific_operating_systems

      Essentially you just need to download HiveMQ and start the run.bat file. The link also contains detailed instructions how to set up HiveMQ as Windows Service.

      Best,
      Dominik from the HiveMQ Team

  13. Andreas says:

    Hello,

    is it possible to connect two MQTT-Broker together? So the Broker 1 is client of the Broker 2 and receive messages from a configured topic. Background: Communication between two party from the DMZ. Both party are receiving and sending data.

    1. Hi Andreas,

      chaining brokers is possible with bridging, which is not a MQTT standard feature, some MQTT broker support it though. We would recommend to go with clustering if possible since bridging in general does not scale as well as clustering.

      All the best,
      Dominik from the HiveMQ Team

  14. Mounika says:

    Awsome work.this is very helpful but what about programming in C. Will u please suggest me some links for MQTT in C.

  15. mahem says:

    Well Explained.

  16. Naly says:

    Amazing article. It provided clear explanations and has helped me greatly for the tasks given at my job. Thank you enormously. :)

  17. Arpan says:

    Thank you very much. The article is very useful and is explained very clearly.
    :)
    Is there any way the broker can cache published messages in a topic and then send them to the subscriber once it comes online? I tried the above example, however, I could not get previous published messages.
    Once I ran the publisher again, while the SubscriberCallback was on, I was able to get only the real time published messages. Am I doing something wrong?

    FYI, I have downloaded the HIVE Broker and running it locally on my machine.

    Thank you once again!

  18. Arpan says:

    Hi its Arpan again.

    Do you have any documentation on PUB/SUB using NodeJS ?

    1. Hi Arpan,

      check out the MQTT Client Library Encyclopedia blog post by Matteo Collina (the author of MQTT.js): http://www.hivemq.com/blog/mqtt-client-library-mqtt-js

      Hope this helps,
      Dominik from the HiveMQ Team

  19. Rich says:

    Great article, and seriously considering MQTT.

    Using your example, I have a few “temperature sensors”, but they must go into hibernation and only wake up when receiving requests to activate from the broker. Is there a way to do this while using very little power via Wi-Fi?

    Thanks for a great writeup!

  20. Hardik says:

    We have been using HiveMQ for long time.Now we are configuring our local environment to have Cluster type environment. Which is best way to implement cluster on linux system(Pi is best for us if possible)…

    Thank you..

    1. Hi Hardik,

      bear in mind that HiveMQ is designed for high performance and Raspberry Pis are not the ideal environment for high performance deployments. You should consider using in-memory persistence if you want to increase the life of your flash drive. If you don’t need elastic clustering, I would go for static cluster discovery (see http://www.hivemq.com/docs/hivemq/latest/#cluster-tcp). If the Raspberry Pis reside in the same network, you can also try UDP multicast discovery or broadcast discovery to enable the elastic cluster capabilities. You can also learn more about the cluster here: http://www.hivemq.com/clustering-hivemq-paper/

      Hope this helps,
      Dominik from the HiveMQ

  21. marwa says:

    Hello,I am a beginner in java and I want to know how the execution of nombrous codesto see and to understand the result can be done and how a connexion between publishers and subscribers can be established

    1. Hi Marwa,

      you can find more information about the Paho MQTT Java client in the following blog post from our MQTT Client encyclopedia: http://www.hivemq.com/blog/mqtt-client-library-encyclopedia-eclipse-paho-java

      Also more general information about how the connection establishment works and about MQTT in general can be found in our MQTT Essentials series (http://www.hivemq.com/mqtt-essentials/)

      Hope this helps,
      Christian from the HiveMQ team

  22. marwa says:

    Hello
    I undestand how can we have a subscriber and a publisher.I want to ask how can we implement many publishers and subscribers in the same machine to test a tiny IOT

  23. Sushmitha says:

    Hello,

    I’m using Spring IDE and I’ve imported paho project (https://github.com/eclipse/paho.mqtt.java) to my workspace.I then created a new class in the package org.eclipse.paho.client.mqttv3.test.The issue is I get an error at Utils.getMacadress saying Utils cannot be resolved.
    KIndly help me with this.PLease let me know if I’m doing things the right way.I would be very thankful if you can explain me where to put the source code you provided and how to test it and from where to download all the dependencies etc.
    Please revert back!

  24. Jay says:

    Hello,
    Great blog. Thank you!
    I am developing a bi-direction communication project in which I would like to send/receive the data (device info., state, level etc.) from a light-weight java application installed on BeagleBone Black to from the cloud-based web application (User interface with database), vice-versa.

    Using login credentials to User interface, user can see their devices’ data and send device specific commands/events (like change state to 0 or move level to 50%) to BBB.
    FYI: Currently, BBB handles all the serial communication with devices using a library configured in a light-weight application.

    My question is can I use MQTT to send payload data from BBB to cloud web application, and vice-versa.
    If yes, is there any tutorials that I can refer to make this happen?
    If no, would you please suggest how can this be possible?

    Thank you in advance!

  25. Dinesh Thogulua says:

    Why is it always the case that whenever I am looking for an article about some MQTT topic, I find badly written articles everywhere except in HiveMQ. Good job. Really appreciate it.

  26. Ethiraj says:

    Very Good article,

  27. Neeraj says:

    Hi,
    Very good explanation. I have one concern. Is it possible to use ARDUINO UNO and ESP8266(WiFi) module for establishing this MQTT connection. If possible then could you please guide me or any helpful link for this.
    Every help is welcomed.
    Thank You

    1. Hi,

      there are some MQTT libraries for Arduino.

      https://github.com/256dpi/arduino-mqtt
      https://github.com/knolleary/pubsubclient

      Hope that helps,
      Christoph from the HiveMQ Team

  28. talamaki says:

    Great article. Thanks!

  29. archana sahu says:

    hello, i m working here on ESP8266 nodemcu board with the micropython langauge. i m trying to use MQTT protocol to let my board communicate with the client(cmd promt). i m using the local IP i.e, 127.0.0.1 : 1883(server) as mosquitto broker for subscribing to some topic and publishing on to the same topic . i have imported all necessary modules required for the communication(umqtt.py or simple.py on github), but when i try to connect my board with c.connect() command , it is giving me an error. it will be helpful if anybody resolve the problem where i m stucking from past few days,
    well thanx in advance.

  30. Ajay says:

    Hello there i am Ajay,
    I want to know how can I Push the data from client to server and vice versa using MQTT by Linux!
    Thanks And Regards.

    1. Hi Ajay,

      check out the MQTT Essentials and the MQTT Client Library Encyclopedia on how to get started with your MQTT project.

      Hope that helps,
      Christoph from the HiveMQ Team

  31. Alec says:

    I had to use the “new MqttClient(String serverURI, String clientId, MqttClientPersistence persistence)” constructor in order to avoid a MqttPersistenceException. I passed in a “new MemoryPersistence()” object and that made the exception go away, in case anyone else runs into that issue.

  32. Mehdi says:

    This is the best introduction which I have seen!
    Thank you HiveMQ

  33. Mohammad Asif says:

    This is the best one!!!
    Thanks a lot!!

  34. timbuktu says:

    Superbly written. A shame that it is in Java, but you guys have given me a much better overview and walkthrough of the topic than any other source out there. Your implementation as a whole looks solid too. Thank you.

  35. alai says:

    Very good article, help me a lot

  36. Vishal says:

    When TCP comes into picture if we run this application?

    1. Hallo,

      TCP is your default socket protocol for MQTT communication.
      We suggest this blog series to get a good grasp of MQTT.

      Hope that helps.

      Regards,
      Florian, from the HiveMQ Team.

  37. RP Singh says:

    We are LAMP (php ) developers how can we connect to MQTT … is it possible … LiveProject Club

    1. Hi there,

      nice to see you’re taking an interest in MQTT.
      The Mosquitto PHP library allows you to connect to an MQTT broker, using PHP.

      Hope that helps.
      Regards,
      Florian, form the HiveMQ Team.

  38. KRISHNAVENI says:

    Hi i am working on MSP 430 microcontroller ,can u suggest me to implemnet MQTT b/w two C-motes

    1. Hi there,

      Die you check out this text manual arleady?
      I am not aware of another library usable with the MSP430.

      Kind regards,
      Florian from the HiveMQ Team.

  39. Dustin says:

    So MQTT is a protocol that uses a publish-subscribe architecture instead of request-response like HTTP does?

    If this is the case, I could implement the collection of sensor data in python/zmq with something similar to this?:
    https://gist.github.com/ramn/7061042

    and use whatever language I want from the control server, as long as it has a zmq library
    http://zguide.zeromq.org/page:all

Leave a Reply

Your email address will not be published. Required fields are marked *