MQTT Essentials Part 2: Publish & Subscribe

MQTT Essentials Banner - Publish / Subscribe

Welcome to the second edition of MQTT Essentials. A ten-part blog series on the core features and concepts of the MQTT protocol. In the first post of this series, we introduced MQTT and talked about the origin and history. If you missed part one, you should definitely check it out

The publish/subscribe pattern

The publish/subscribe pattern (also known as pub/sub) provides an alternative to traditional client-server architecture. In the client-sever model, a client communicates directly with an endpoint.The pub/sub model decouples the client that sends a message (the publisher) from the client or clients that receive the messages (the subscribers). The publishers and subscribers never contact each other directly. In fact, they are not even aware that the other exists. The connection between them is handled by a third component (the broker). The job of the broker is to filter all incoming messages and distribute them correctly to subscribers. So, let’s dive a little deeper into some of the general aspects of pub/sub (we’ll talk about MQTT specifics in a minute).

MQTT Publish / Subscribe

MQTT Publish / Subscribe

The most important aspect of pub/sub is the decoupling of the publisher of the message from the recipient(subscriber). This decoupling has several dimensions:

  • Space decoupling: Publisher and subscriber do not need to know each other (for example, no exchange of IP address and port).

  • Time decoupling: Publisher and subscriber do not need to run at the same time.

  • Synchronization decoupling: Operations on both components do not need to be interrupted during publishing or receiving.

In summary, the pub/sub model removes direct communication between the publisher of the message and the recipient/subscriber. The filtering activity of the broker makes it possible to control which client/subscriber receives which message. The decoupling has three dimensions: space, time, and synchronization.

Scalability

Pub/Sub scales better than the traditional client-server approach. This is because operations on the broker can be highly parallelized and messages can be processed in an event-driven way. Message caching and intelligent routing of messages are often a decisive factors for improving scalability. Nonetheless, scaling up to millions of connections is a challenge. Such a high level of connections can be achieved with clustered broker nodes to distribute the load over more individual servers using load balancers. (this topic is beyond the scope of the current article, we’ll cover it in a separate post).

Message filtering

It’s clear that the broker plays a pivotal role in the pub/sub process. But how does the broker manage to filter all the messages so that each subscriber receives only messages of interest? As you’ll see, the broker has several filtering options:

Option 1: Subject-based filtering

This filtering is based on the subject or topic that is part of each message. The receiving client subscribes to the broker for topics of interest. From that point on, the broker ensures that the receiving client gets all message published to the subscribed topics. In general, topics are strings with a hierarchical structure that allow filtering based on a limited number of expressions.

Option 2: Content-based filtering

In content-based filtering, the broker filters the message based on a specific content filter-language. The receiving clients subscribe to filter queries of messages for which they are interested. A significant downside to this method is that the content of the message must be known beforehand and cannot be encrypted or easily changed.

Option 3: Type-based filtering

When object-oriented languages are used, filtering based on the type/class of a message (event) is a common practice. For example,, a subscriber can listen to all messages, which are of type Exception or any sub-type.

Of course, publish/subscribe is not the answer for every use case. There are a few things you need to consider before you use this model. The decoupling of publisher and subscriber, which is the key in pub/sub, presents a few challenges of its own. For example, you need to be aware of how the published data is structured beforehand. For subject-based filtering, both publisher and subscriber need to know which topics to use. Another thing to keep in mind is message delivery. The publisher can’t assume that somebody is listening to the messages that are sent. In some instances, it is possible that no subscriber reads a particular message.

MQTT

Now that we’ve explored the publish/subscribe model in general, let’s focus on MQTT specifically. Depending on what you want to achieve, MQTT embodies all the aspects of pub/sub that we’ve mentioned:

  • MQTT decouples the publisher and subscriber spatially. To publish or receive messages, publishers and subscribers only need to know the hostname/IP and port of the broker

  • MQTT decouples by time. Although most MQTT use cases deliver messages in near-real time, if desired, the broker can store messages for clients that are not online. (Two conditions must be met to store messages: the client had connected with a persistent session and subscribed to a topic with a Quality of Service greater than 0).

  • MQTT works asynchronously. Because most client libraries work asynchronously and are based on callbacks or a similar model, tasks are not blocked while waiting for a message or publishing a message. In certain use cases synchronization is desirable and possible. To wait for a certain message, some libraries have synchronous APIs. But the flow is usually asynchronous.

Another thing that should be mentioned is that MQTT is especially easy to use on the client-side. Most pub/sub systems have the logic on the broker-side, but MQTT is really the essence of pub/sub when using a client library and that makes it a light-weight protocol for small and constrained devices.

MQTT uses subject-based filtering of messages. Every message contains a topic (subject) that the broker can use to determine whether a subscribing client gets the message or not. See part 5 of MQTT Essentials to learn more about the concept of topics. If desired, you can also set up content-based filtering with the HiveMQ MQTT broker and our custom plugin system.

To handle the challenges of a pub/sub system, MQTT has quality of service (QoS) levels. You can easily specify that a message gets successfully delivered from the client to the broker or from the broker to a client. But there is the chance that nobody subscribes to the particular topic. If this is a problem, it depends on the broker how to handles such cases. For example, the HiveMQ MQTT broker has a plugin system, which can identifying such cases. You can either have the broker take action or simply log every message into a database for historical analyses. To keep the hierarchical topic tree flexible , it is important to design the topic tree very carefully and leave room for future use cases. If you follow these strategies, MQTT is perfect for production setups.

Distinction from message queues

There is a lot of confusion about the name MQTT and whether the protocol is implemented as a message queue or not. We will try to shed some light on the topic and explain the differences. In ourlast post, we mentioned that MQTT refers to the MQseries product from IBM and has nothing to do with “message queue“. Regardless of where the name comes form, it’s useful to understand the differences between MQTT and a traditional message queue:

A message queue stores message until they are consumed
When you use a message queue, each incoming message is stored in the queue until it is picked up by a client (often called a consumer). If no client picks up the message, the message remains stuck in the queue and waiting to be consumed. In a message queue, it is not possible for a message not to be processed by any client, like it is in MQTT if nobody subscribes to a topic.

A message will only be consumed by one client
Another big difference is that in a traditional message queue a message can be processed by one consumer only. The load is distributed between all consumers for a queue. In MQTT the behavior is quite the opposite: every subscriber that subscribes to the topic gets the message.

Queues are named and must be created explicitly
A queue is far more rigid than a topic. Before a queue can be used, the queue must be created explicitly with a separate command. Only after the queue is named and created is it possible to publish or consume messages. In contrast, MQTT topics are extremely flexible and can be created on the fly.

If you can think of any other differences that we overlooked, we would love to hear from you in the comments.


That’s the end of post number two of MQTT Essentials. Next week, we are going to look more closely into what makes a MQTT client and broker and how the two connect.

To get the next part of this series as soon as it is released, enter your email in the subscription form below. If you prefer RSS, you can subscribe to our RSS feed here.

59 comments

  1. Carlos Santes says:

    Great, thaks for the effort, I’m enjoying the ride.

  2. Ahmed Al-Haddad says:

    This is actually some great work. Kudos!

  3. Raj says:

    Hi,
    Your tutorial is great. I am following the step mentioned in your tutorial. I have one problem while publishing and subscribing.

    In my application I have two instance of MqttClient with different username, password and clientId but with the same URL. Problem which I am facing is when I connect to first client then second client got disconnect and if I connect second client then first got disconnect. what kind of behavior is this. both client do not connect at the same time.

    If this is the behavior of Mqtt and what is the alternative.

    Please suggest.

    1. Hi Raj,

      thanks for the feedback. Can you double check that you don’t use the same client identifier for both clients? In case you’re using the same client id, a client takeover occurs and one of the clients gets kicked out.

      Hope this helps,
      Dominik from the HiveMQ Team

  4. Jay says:

    Great Series of Articles! Thanks a lot.

    I have a question though, how does a subscriber get to know about the topics available with the broker? I didn’t see any information as to how a subscriber can decide which all topics to subscribe based on the list of available topics. Does the MQTT protocol provide a provision for the same?

    1. Hi Jay,

      great question!
      The subscriber does not know which topics are available. We have seen many deployments where far more than 100.000+ topics are used in the system and sending these topic lists to subscribers would be overkill. The protocol itself does not define such a topic discovery. We have seen these two popular solutions in deployments:

      * Subscribe to a custom $SYS topic which contains all available topics for the client (e.g. $SYS/{client_id}/topics). You can learn about hooking in custom $SYS topics here: http://www.hivemq.com/docs/plugins/latest/#systopic-service
      * Create a REST Resource with HiveMQs REST Service which allows discovery over HTTP. You can learn about the RESTService here: http://www.hivemq.com/docs/plugins/latest/#rest-service

      Hope this helps,
      Dominik from the HiveMQ Team.

  5. Jay says:

    Hi HiveMQ Team (Dominik),

    Thanks a lot for your reply.

    You have mentioned one approach as:
    ” Subscribe to a custom $SYS topic which contains all available topics for the client (e.g. $SYS/{client_id}/topics). You can learn about hooking in custom $SYS topics here: http://www.hivemq.com/docs/plugins/latest/#systopic-service

    — In this case, the “client_id” refers to the subscribers client-id.. right? By default, are all topics not available to all subscribers?

    1. Hi Jay,

      by default (if no authorization logic is in place), every MQTT client can subscribe to every topic. What I mentioned was a business logic (which client can subscribe to which topic) which needs to be implemented by the plugin for the specific use case when using the $SYS topic approach. So the client identifier is the subscriber client identifier from the last example.

      One note regarding the “available topics”: MQTT uses dynamic topics by default, so there can be an arbitrary (enormous) number of topics in the system because every MQTT client can use any MQTT topic for publishing and subscribing.

      Hope this helps,
      Dominik from the HiveMQ Team

  6. vj_time says:

    Great tutorials! thank you very much! can you please publish a series for implementation of MQTT….with source codes….such that we can learn from it and use it in real time. that would be a great help!

    1. Hi Vishal,

      there is a blog post series about MQTT clients for a bunch of different languages available here: http://www.hivemq.com/mqtt-client-library-encyclopedia

      Thanks so much for your feedback,
      Dominik from the HiveMQ Team

  7. sc says:

    I’m getting a ‘404 – Page Not Found’ when attempting to access Part 1
    (http://www.hivemq.com/blog/mqtt-essentials-part-1-introducing-mqtt)

    1. Hi,

      the link seems to work from here. Does this problem still happen? If the problem persists, can you share more details?

      Thanks,
      Dominik from the HiveMQ Team

  8. Abhinaya says:

    Hi , I have a doubt when i’m publishing the messages, receiver not receiving that message but broker sends ACK to me(sender) message was successfully delivered. How can I know the message reached to receiver or not?

    1. Hi Abhinaya,

      that is a great question. You won’t get notified when the receiver (or better: the receivers) received the message from the Broker you just sent to the Broker. If you would like to get notified, you must implement such a mechanism yourself on top of MQTT (e.g. by sending messages to different topics).

      Hope this helps,
      Dominik from the HiveMQ Team

  9. Salil says:

    Hi,

    I am a little confused about the use of MQTT Client libraries. How and where should they be used?
    Example: Let’s say I have an IoT device (e.g. Thermostat). I want this IoT device (MQTT Client) to communicate (publish) its temperature to a broker. The broker will in turn send this temperature to another MQTT client. How do I make the IoT device communicate using the MQTT protocol? Is this the place where MQTT client libraries are used?

    1. Hi Salil,

      it is exactly as you described.
      In your case all your IoT devices are MQTT clients.
      For example one device is a publisher and one is a subscriber.
      They can communicate by using the same topic.

      Best regards,
      Christian from the HiveMQ Team

  10. Nandan G says:

    Hi,
    I am facing an issue with QOS1 and QOS2 implementation as a publisher client.
    My implementation is made as synchronous,it will not send the second message until first message handshake completes.
    1)PUBLISH ->
    2)PUBREC
    4)PUBCOMP<-
    If the acknowledgement is not recieved within in 10 seconds, application is sending back the same message with duplicate flag.(PUBLISH or PUBREL)
    But sometimes, the broker is closing the connection when i send the duplicate request. why is it so?
    I have observed one thing, while after i'll send the duplicate request, i have been recieving the previously sent acknowledgement(I mean to say, same request is being sent twice with duplicate flag and I am getting same ack twice). Will it cause any problem?
    How long i should wait for the acknowledgement at publisher client side?

    1. Nandan G says:

      Sequence is like this,
      1)PUBLISH ->
      2)PUBREC
      4)PUBCOMP<-

    2. Hi Nandan,

      you can set HiveMQ into debug mode to see why your client gets disconnected.

      See the documentation for more information:
      http://www.hivemq.com/docs/hivemq/latest/#hivemqdocs_logging

      Hope that helps,
      Christoph from the HiveMQ team.

  11. Nancy says:

    Hi,
    I am facing an issue about adding topic , then how to notify multiple clients can listener this event?

  12. Rafael Pimenta says:

    Great job! That’s very useful for a lot of people!

    Best Regards

  13. Ivan says:

    Hy,
    link for content-based filtering doesnt work: (“…It would also be possible to do content-based filtering with the HiveMQ MQTT broker and its custom plugin system….”)

    1. Hi Ivan,

      thank you for pointing that out. It is fixed now.

      Regards,
      Florian from the HiveMQ Team

  14. Toi Nguyen says:

    The HiveMQ Team,
    Can the publisher get the notification when the subcribers have been received messages ?

    1. Hi Toi,

      the behavior you are describing is not part of MQTT. You can use Quality of Service> greater than 0 to ensure, a message is being published. If you want an acknowledgement, that the message has been received by the subscriber, you need to implement the functionality yourself.
      HiveMQ’s Plug In System offers the ability to implement your own logic.

      Hope that was helpful,
      Florian of the HiveMQ Team.

  15. Ajit says:

    What happens if there are topics of same name?? Is there any mechanism to differentiate between them?

    1. Hallo Ajit,

      There can only be one topic with the same name. A topic is defined only by it’s name. If you want to differentiate which subscribers get specific messages, you need to publish them on separate topics.

      I hope that helps.
      Regards,
      Florian, from the HiveMQ Team.

  16. Alex says:

    I’m struggling to understand how you would set up a chat room of 5 participants in which all 5 are able to receive and send messages from/to the room, but without having the broker send messages back to the initial sender? If user 1 sens a message it should be broadcasted to user 2, 3, 4 and 5 but not to user 1 (since that client obviously already has the data). While still having user 1 able to receive messages from the other users.

    1. Hallo Alex,

      There are many ways to achieve your described behaviour.
      For example you could have every user publish on their own individual topic, while everyone subscribes to the individual topics for every user in the chat room.

      Kind regards,
      Florian from The HiveMQ Team.

  17. aditya says:

    hi i will make mqtt with ip camera to send the video streaming over mqtt protocol, it’s possible?

    1. Hallo Aditya,

      In theory it is possible to use MQTT for streaming but that is not what the protocol is intended for and we would strongly recommend not using MQTT for streaming.

      Kind regards,
      Florian from The HiveMQ Team.

  18. Anna says:

    Dear HiveMW Team,

    thanks for your amazing thread!

    I got confused with something… I understood that it’s possible for a client X to subscribe to more than one topic. But, is it possible for a client Y to publish on more topics ? Or could we have many clients publishing on same topic ? Could you maybe provide some practical examples, please?

    Thanks ! And keep up the amazing work!

    Anna

    1. Hallo Anna,

      Glad to see you are taking an interest in MQTT and are liking our post blog series.
      To answer your question:
      Yes it is possible for a client to publish on multiple topics. It is also possible for multiple clients to publish on the same topic.
      For example imagine a number of sensors collecting data, each periodically publishing on their own individual topics. (Think something like company/area/deviceId/metric)
      You could at the same time have some sort of global alert topic, that all the devices publish on if a threshold is being crossed.

      I hope this clarifies things for you.

      Kind regards,
      Florian from the HiveMQ Team.

  19. Jose Carlos Moraes says:

    Thanks for helping us and introducing a better understanding of MQTT. Great job.
    Thanks and regards

  20. Samuel says:

    Dear HiveMQ team,

    Sequel to Anna’ s question below:

    I got confused with something… I understood that it’s possible for a client X to subscribe to more than one topic. But, is it possible for a client Y to publish on more topics ? Or could we have many clients publishing on same topic ? Could you maybe provide some practical examples, please?

    Thanks ! And keep up the amazing work!

    Anna

    I would like to know if: A publisher (with a client Id) can be a subscriber? That is,

    1. The publisher publishes some topics whilst it subscribes to other topics which it doesn’t have – using same client_id?
    2. Or needs another client_id to be able to subscribe to topic?
    3. Does the configuration fields allow for multiple Client_Id?

    Thank you,

    Samuel

    1. Hallo Samuel,

      nice to see you are taking an interest in MQTT.
      A single client can be both subscriber and publisher. Both operations are completely independent from one another.
      You can also use a single clientID per client.

      Kind regards,
      Florian from The HiveMQ Team.

  21. afsar says:

    sub client sub. for topic and multiple clients publish for same topic, so how i’ll handle receiving multiple msg same time ?

    1. Hallo Asfar,

      Nice to see your taking an interest in MQTT and HiveMQ.
      The broker will only forward one message on the same topic at the time, so you won’t receive multiple messages at the same time.
      Hope this answers your question.

      Kind regards,
      Florian

  22. Jiangnan says:

    Dear HiveMQ team,
    Can you give me an example to explain what is “Synchronization decoupling”?
    Thank you,
    Xiang

    1. HI Xiang,

      Nice to see you are taking an interest in MQTT and HiveMQ.
      Let me try to illustrate what synchronization decoupling is:
      Actions for 2 or more clients that are connected to the same broker are not blocked by each other.
      More specifically.
      Client 1 can send data to the broker at the same time client 2 can receive data from the broker.

      I hope this answers your question.

      Kind regards,
      Florian from the HiveMQ Team.

  23. vik says:

    client.subscribe(‘#’) means? what # ?

    1. Hi Vik,

      Nice to see you are taking an interest in MQTT and HiveMQ.
      The ‘#’ is a so called Wildcard.You can find detailed descriptions in the best practises part of this blog series.

      Kind regards,
      Florian

  24. Timothy Lynch says:

    Just a heads up: there’s a grammar typo in the very first paragraph: “and then we’ll focusing on MQTT”. This could be corrected by changing it to “and then we’ll focus on MQTT” or “and then we’ll be focusing on MQTT”.

    1. Hi Timothy,

      Thank you for pointing this out. I corrected it 🙂

      Kind regards,
      Florian from The HiveMQ Team.

  25. Dehan Louw says:

    This is fantastically written. So often, us technical people fail to hit the balance of complexity- explaining either too simplified or (more often) failing to grasp the knowledge context of the new reader.

    I thoroughly enjoyed learning about the topic from no background knowledge

    Thanks you

    1. Hi Dehan,

      Thanks a lot for the kind words, we appreciate them.

      Kind regards,
      Abdullah from The HiveMQ Team.

  26. proxy ahmed says:

    hello

    is it possible to transfer voice/sound file(mp3/wav etc) using mqtt protocol? if possible, how can i publish sound file to broker and how to receive that sound file from subscriber end as a sound file ?

    if possible plz share a android app that transfer sound file through mqtt so that i can understand the coding process.

    1. Hi Ahmed,

      Nice to see you’re taking an interest in MQTT and HiveMQ.

      MQTT is a data agnostic protocol that can transfer any kind of information. For that sound files like in your case need to be first converted in byte array before the publish. The subscriber (receiver) needs than to convert back the received byte array.

      There is a client library for android with examples which you can try out.

      Kind regards,
      Abdullah from the HiveMQ Team

  27. muna says:

    Hi HiveMQ team,
    Thanks for this great tutorial, I have one comment please about theses sentence
    “Therefore some libraries have synchronous APIs in order to wait for a certain message. But usually the flow is asynchronous. ” I cannot get it very well. Could you give me a real example of this issue? and how is the flow is still asynchronous even the messages are synchronous.?!

    1. Hi Muna,

      thank you for the kind words.

      You need to know that MQTT has an asynchronous message flow which means that your program doesn’t need to wait actively for a response and therefore isn’t blocked in the execution (synchronization decoupling).

      Though client libraries may offer synchronous APIs too which are actively waiting for the response and are blocking the execution of your program until you receive a response. The advantage of this approach is however that you can create fast and easy to read proof of concepts.

      Kind regards,
      Abdullah from the HiveMQ Team

  28. Andrei says:

    There’s a small wording error I think
    “Pub/Sub decouples a client, who is sending a particular message (called publisher) from another client (or more clients)”
    Shouldn’t instead of “from another client” be “TO another client” ?

    1. Hi Andrei,

      thank you for the feedback.

      With that sentence we’re pointing out that a client decouples himself from the receiving client(s). That means that a client doesn’t need to know anything from the another client.

      Hope I could clear it up.

      Kind regards,
      Abdullah from the HiveMQ Team.

  29. Valery says:

    Hello!

    How to set up mqttt username and password? Should i register them somewhere or i just publish them? If so what is the sense of such authentification?

    1. Hi Valery,

      Nice to see you’re taking an interest in MQTT and HiveMQ.

      HiveMQ doesn’t require by default any authentication. You could though create a plugin for HiveMQ which implements the check for a valid username and password.

      Kind regards,
      Abdullah from the HiveMQ Team.

Leave a Reply

Your email address will not be published. Required fields are marked *