MQTT Essentials Part 2: Publish & Subscribe

MQTT Essentials Banner - Publish / Subscribe

Welcome to the second part of the MQTT Essentials. A blog series about the core features and concepts in the MQTT protocol. In this second post, we’ll discuss the Publish/Subscribe pattern. First we look at the general characteristics of Publish/Subscribe itself and then we’ll focusing on MQTT. We’ll also explain how MQTT is different from traditional message queuing protocols.

In the first post, we introduced MQTT, explained its origin and history. If you haven’t already read it, you should definitely check it out.

The publish/subscribe pattern

The publish/subscribe pattern (pub/sub) is an alternative to the traditional client-server model, where a client communicates directly with an endpoint. However, Pub/Sub decouples a client, who is sending a particular message (called publisher) from another client (or more clients), who is receiving the message (called subscriber). This means that the publisher and subscriber don’t know about the existence of one another. There is a third component, called broker, which is known by both the publisher and subscriber, which filters all incoming messages and distributes them accordingly. So let’s dive into a little bit more details about the just mentioned aspects. Remember this is still the basic part about pub/sub in general, we’ll talk about MQTT specifics in just a minute.

MQTT Publish / Subscribe

MQTT Publish / Subscribe

As already mentioned the main aspect in pub/sub is the decoupling of publisher and receiver, which can be differentiated in more dimensions:

  • Space decoupling: Publisher and subscriber do not need to know each other (by ip address and port for example)

  • Time decoupling: Publisher and subscriber do not need to run at the same time.

  • Synchronization decoupling: Operations on both components are not halted during publish or receiving

In summary publish/subscribe decouples publisher and receiver of a message, through filtering of the messages it is possible that only certain clients receive certain messages. The decoupling has three dimensions: Space, Time, Synchronization.

Scalability

Pub/Sub also provides a greater scalability than the traditional client-server approach. This is because operations on the broker can be highly parallelized and processed event-driven. Also often message caching and intelligent routing of messages is decisive for improving the scalability. But it is definitely a challenge to scale publish/subscribe to millions of connections. This can be achieved using clustered broker nodes in order to distribute the load over more individual servers with load balancers. (We will discuss this in detail in a separate post, this would go beyond the scope).

Message Filtering

So what’s interesting is, how does the broker filter all messages, so each subscriber only gets the messages it is interested in?

Option 1: Subject-based filtering

The filtering is based on a subject or topic, which is part of each message. The receiving client subscribes on the topics it is interested in with the broker and from there on it gets all message based on the subscribed topics. Topics are in general strings with an hierarchical structure, that allow filtering based on a limited number of expression.

Option 2: Content-based filtering

Content-based filtering is as the name already implies, when the broker filters the message based on a specific content filter-language. Therefore clients subscribe to filter queries of messages they are interested in. A big downside to this is, that the content of the message must be known beforehand and can not be encrypted or changed easily.

Option 3: Type-based filtering

When using object-oriented languages it is a common practice to filter based on the type/class of the message (event). In this case a subscriber could listen to all messages, which are from type Exception or any subtype of it.

Of course publish/subscribe is not a silver bullet and there are some things to consider, before using it. The decoupling of publisher and subscriber, which is the key in pub/sub, brings a few challenges with it. You have to be aware of the structuring of the published data beforehand. In case of subject-based filtering, both publisher and subscriber need to know about the right topics to use. Another aspect is the delivery of message and that a publisher can’t assume that somebody is listening to the messages he sends. Therefore it could be the case that a message is not read by any subscriber.

MQTT

So now we have learned a lot about publish/subscribe in general, but what about MQTT in specific. MQTT embodies all of the mentioned aspects, depending on what you want to achieve with it. MQTT decouples the space of publisher and subscriber. So they just have to know hostname/ip and port of the broker in order to publish/subscribe to messages. It also decouples from time, but often this is just a fall-back behavior, because the use case mostly is to delivery message in near-realtime. Of course the broker is able to store messages for clients that are not online. (This requires two conditions: client has connected once and its session is persistent and it has subscribed to a topic with Quality of Service greater than 0). MQTT is also able to decouple the synchronization, because most client libraries are working asynchronously and are based on callbacks or similar model. So it won’t block other tasks while waiting for a message or publishing a message. But there are certain use cases where synchronization is desirable and also possible. Therefore some libraries have synchronous APIs in order to wait for a certain message. But usually the flow is asynchronously. Another thing that should be mentioned is that MQTT is especially easy to use on the client-side. Most pub/sub systems have the most logic on the broker-side, but MQTT is really the essence of pub/sub when using a client library and that makes it a light-weight protocol for small and constrained devices.

MQTT uses subject-based filtering of messages. So each message contains a topic, which the broker uses to find out, if a subscribing client will receive the message or not. More details about topics are explained in Part 5 of the MQTT Essentials. It would also be possible to do content-based filtering with the HiveMQ MQTT broker and its custom plugin system.

In order to handle the challenges of a pub/sub system in general, MQTT has the quality of service (QoS) levels. It is easily possible to specify that a message gets successfully delivered from the client to the broker or from the broker to a client. But still there is the chance that nobody subscribes to the particular topic. If this is a problem, it depends on the broker how to handles such cases. For example, the HiveMQ MQTT broker has a plugin system, which is capable of identifying such cases and take action or just log every message into a database for historical analytics. In order to mitigate the inflexibility of topics, it is important to design the topic tree very carefully and leave room for extension for use cases in the future. If you follow these strategies, MQTT is perfect for production setups.

Distinction from Message Queues

So there are many confusions about MQTT, its name and if it is implemented as a message queue or not. We will try to bring light into the dark and explain the differences. In our last post we already pointed out that the name MQTT comes from an IBM product called MQseries and has nothing to do with “message queue“. But regardless of the name, what are the differences between MQTT and a traditional message queue?

A message queue stores message until they are consumed
When using message queues, each incoming message will be stored on that queue until it is picked up by any client (often called consumer). Otherwise the message will just be stuck in the queue and waits for getting consumed. It is not possible that message are not processed by any client, like it is in MQTT if nobody subscribes to a topic.

A message will only be consumed by one client
Another big difference is the fact that in a traditional queue a message is processed by only one consumer. So that the load can be distributed between all consumers for a particular queue. In MQTT it is quite the opposite, every subscriber gets the message, if they subscribed to the topic.

Queues are named and must be created explicitly
A queue is far more inflexible than a topic. Before using a queue it has to be created explicitly with a separate command. Only after that it is possible to publish or consume messages. In MQTT topics are extremely flexible and can be created on the fly.

If there are any other differences that we have missed, we would love your opinion in the comments.


So that’s the end of post number two and next week we will look more into the definitions of a MQTT client and broker as well as establishing a connection between them.

If you want to be notified as soon as the next part is released simply sign up for our newsletter, which brings you fresh content about MQTT and HiveMQ once a week. If you prefer RSS, you can subscribe to our RSS feed here.

Sources

38 comments

  1. Carlos Santes says:

    Great, thaks for the effort, I’m enjoying the ride.

  2. Ahmed Al-Haddad says:

    This is actually some great work. Kudos!

  3. Raj says:

    Hi,
    Your tutorial is great. I am following the step mentioned in your tutorial. I have one problem while publishing and subscribing.

    In my application I have two instance of MqttClient with different username, password and clientId but with the same URL. Problem which I am facing is when I connect to first client then second client got disconnect and if I connect second client then first got disconnect. what kind of behavior is this. both client do not connect at the same time.

    If this is the behavior of Mqtt and what is the alternative.

    Please suggest.

    1. Hi Raj,

      thanks for the feedback. Can you double check that you don’t use the same client identifier for both clients? In case you’re using the same client id, a client takeover occurs and one of the clients gets kicked out.

      Hope this helps,
      Dominik from the HiveMQ Team

  4. Jay says:

    Great Series of Articles! Thanks a lot.

    I have a question though, how does a subscriber get to know about the topics available with the broker? I didn’t see any information as to how a subscriber can decide which all topics to subscribe based on the list of available topics. Does the MQTT protocol provide a provision for the same?

    1. Hi Jay,

      great question!
      The subscriber does not know which topics are available. We have seen many deployments where far more than 100.000+ topics are used in the system and sending these topic lists to subscribers would be overkill. The protocol itself does not define such a topic discovery. We have seen these two popular solutions in deployments:

      * Subscribe to a custom $SYS topic which contains all available topics for the client (e.g. $SYS/{client_id}/topics). You can learn about hooking in custom $SYS topics here: http://www.hivemq.com/docs/plugins/latest/#systopic-service
      * Create a REST Resource with HiveMQs REST Service which allows discovery over HTTP. You can learn about the RESTService here: http://www.hivemq.com/docs/plugins/latest/#rest-service

      Hope this helps,
      Dominik from the HiveMQ Team.

  5. Jay says:

    Hi HiveMQ Team (Dominik),

    Thanks a lot for your reply.

    You have mentioned one approach as:
    ” Subscribe to a custom $SYS topic which contains all available topics for the client (e.g. $SYS/{client_id}/topics). You can learn about hooking in custom $SYS topics here: http://www.hivemq.com/docs/plugins/latest/#systopic-service

    — In this case, the “client_id” refers to the subscribers client-id.. right? By default, are all topics not available to all subscribers?

    1. Hi Jay,

      by default (if no authorization logic is in place), every MQTT client can subscribe to every topic. What I mentioned was a business logic (which client can subscribe to which topic) which needs to be implemented by the plugin for the specific use case when using the $SYS topic approach. So the client identifier is the subscriber client identifier from the last example.

      One note regarding the “available topics”: MQTT uses dynamic topics by default, so there can be an arbitrary (enormous) number of topics in the system because every MQTT client can use any MQTT topic for publishing and subscribing.

      Hope this helps,
      Dominik from the HiveMQ Team

  6. vj_time says:

    Great tutorials! thank you very much! can you please publish a series for implementation of MQTT….with source codes….such that we can learn from it and use it in real time. that would be a great help!

    1. Hi Vishal,

      there is a blog post series about MQTT clients for a bunch of different languages available here: http://www.hivemq.com/mqtt-client-library-encyclopedia

      Thanks so much for your feedback,
      Dominik from the HiveMQ Team

  7. sc says:

    I’m getting a ‘404 – Page Not Found’ when attempting to access Part 1
    (http://www.hivemq.com/blog/mqtt-essentials-part-1-introducing-mqtt)

    1. Hi,

      the link seems to work from here. Does this problem still happen? If the problem persists, can you share more details?

      Thanks,
      Dominik from the HiveMQ Team

  8. Abhinaya says:

    Hi , I have a doubt when i’m publishing the messages, receiver not receiving that message but broker sends ACK to me(sender) message was successfully delivered. How can I know the message reached to receiver or not?

    1. Hi Abhinaya,

      that is a great question. You won’t get notified when the receiver (or better: the receivers) received the message from the Broker you just sent to the Broker. If you would like to get notified, you must implement such a mechanism yourself on top of MQTT (e.g. by sending messages to different topics).

      Hope this helps,
      Dominik from the HiveMQ Team

  9. Salil says:

    Hi,

    I am a little confused about the use of MQTT Client libraries. How and where should they be used?
    Example: Let’s say I have an IoT device (e.g. Thermostat). I want this IoT device (MQTT Client) to communicate (publish) its temperature to a broker. The broker will in turn send this temperature to another MQTT client. How do I make the IoT device communicate using the MQTT protocol? Is this the place where MQTT client libraries are used?

    1. Hi Salil,

      it is exactly as you described.
      In your case all your IoT devices are MQTT clients.
      For example one device is a publisher and one is a subscriber.
      They can communicate by using the same topic.

      Best regards,
      Christian from the HiveMQ Team

  10. Nandan G says:

    Hi,
    I am facing an issue with QOS1 and QOS2 implementation as a publisher client.
    My implementation is made as synchronous,it will not send the second message until first message handshake completes.
    1)PUBLISH ->
    2)PUBREC
    4)PUBCOMP<-
    If the acknowledgement is not recieved within in 10 seconds, application is sending back the same message with duplicate flag.(PUBLISH or PUBREL)
    But sometimes, the broker is closing the connection when i send the duplicate request. why is it so?
    I have observed one thing, while after i'll send the duplicate request, i have been recieving the previously sent acknowledgement(I mean to say, same request is being sent twice with duplicate flag and I am getting same ack twice). Will it cause any problem?
    How long i should wait for the acknowledgement at publisher client side?

    1. Nandan G says:

      Sequence is like this,
      1)PUBLISH ->
      2)PUBREC
      4)PUBCOMP<-

    2. Hi Nandan,

      you can set HiveMQ into debug mode to see why your client gets disconnected.

      See the documentation for more information:
      http://www.hivemq.com/docs/hivemq/latest/#hivemqdocs_logging

      Hope that helps,
      Christoph from the HiveMQ team.

  11. Nancy says:

    Hi,
    I am facing an issue about adding topic , then how to notify multiple clients can listener this event?

  12. Rafael Pimenta says:

    Great job! That’s very useful for a lot of people!

    Best Regards

  13. Ivan says:

    Hy,
    link for content-based filtering doesnt work: (“…It would also be possible to do content-based filtering with the HiveMQ MQTT broker and its custom plugin system….”)

    1. Hi Ivan,

      thank you for pointing that out. It is fixed now.

      Regards,
      Florian from the HiveMQ Team

  14. Toi Nguyen says:

    The HiveMQ Team,
    Can the publisher get the notification when the subcribers have been received messages ?

    1. Hi Toi,

      the behavior you are describing is not part of MQTT. You can use Quality of Service> greater than 0 to ensure, a message is being published. If you want an acknowledgement, that the message has been received by the subscriber, you need to implement the functionality yourself.
      HiveMQ’s Plug In System offers the ability to implement your own logic.

      Hope that was helpful,
      Florian of the HiveMQ Team.

  15. Ajit says:

    What happens if there are topics of same name?? Is there any mechanism to differentiate between them?

    1. Hallo Ajit,

      There can only be one topic with the same name. A topic is defined only by it’s name. If you want to differentiate which subscribers get specific messages, you need to publish them on separate topics.

      I hope that helps.
      Regards,
      Florian, from the HiveMQ Team.

  16. Alex says:

    I’m struggling to understand how you would set up a chat room of 5 participants in which all 5 are able to receive and send messages from/to the room, but without having the broker send messages back to the initial sender? If user 1 sens a message it should be broadcasted to user 2, 3, 4 and 5 but not to user 1 (since that client obviously already has the data). While still having user 1 able to receive messages from the other users.

    1. Hallo Alex,

      There are many ways to achieve your described behaviour.
      For example you could have every user publish on their own individual topic, while everyone subscribes to the individual topics for every user in the chat room.

      Kind regards,
      Florian from The HiveMQ Team.

  17. aditya says:

    hi i will make mqtt with ip camera to send the video streaming over mqtt protocol, it’s possible?

    1. Hallo Aditya,

      In theory it is possible to use MQTT for streaming but that is not what the protocol is intended for and we would strongly recommend not using MQTT for streaming.

      Kind regards,
      Florian from The HiveMQ Team.

  18. Anna says:

    Dear HiveMW Team,

    thanks for your amazing thread!

    I got confused with something… I understood that it’s possible for a client X to subscribe to more than one topic. But, is it possible for a client Y to publish on more topics ? Or could we have many clients publishing on same topic ? Could you maybe provide some practical examples, please?

    Thanks ! And keep up the amazing work!

    Anna

    1. Hallo Anna,

      Glad to see you are taking an interest in MQTT and are liking our post blog series.
      To answer your question:
      Yes it is possible for a client to publish on multiple topics. It is also possible for multiple clients to publish on the same topic.
      For example imagine a number of sensors collecting data, each periodically publishing on their own individual topics. (Think something like company/area/deviceId/metric)
      You could at the same time have some sort of global alert topic, that all the devices publish on if a threshold is being crossed.

      I hope this clarifies things for you.

      Kind regards,
      Florian from the HiveMQ Team.

Leave a Reply

Your email address will not be published. Required fields are marked *