MQTT Essentials Part 5: MQTT Topics & Best Practices

mqttessentials_part5

Welcome to the fifth part of the MQTT Essentials, a blog series about the core features and concepts in the MQTT protocol. In this post we’ll focus on MQTT topics and best practices. As we have already mentioned, topics are used to decide on the MQTT broker which client receive which message. We will also discuss SYS-topics, which are special ones that reveal broker internal information. So let’s get started.

Topics

A topic is a UTF-8 string, which is used by the broker to filter messages for each connected client. A topic consists of one or more topic levels. Each topic level is separated by a forward slash (topic level separator).

topic_basics

In comparison to a message queue a topic is very lightweight. There is no need for a client to create the desired topic before publishing or subscribing to it, because a broker accepts each valid topic without any prior initialization.

Here are a few example topics:

myhome/groundfloor/livingroom/temperature
USA/California/San Francisco/Silicon Valley
5ff4a2ce-e485-40f4-826c-b1a5d81be9b6/status
Germany/Bavaria/car/2382340923453/latitude

Noticeable is that each topic must have at least 1 character to be valid and it can also contain spaces. Also a topic is case-sensitive, which makes myhome/temperature and MyHome/Temperature two individual topics. Additionally the forward slash alone is a valid topic, too.

Wildcards

When a client subscribes to a topic it can use the exact topic the message was published to or it can subscribe to more topics at once by using wildcards. A wildcard can only be used when subscribing to topics and is not permitted when publishing a message. In the following we will look at the two different kinds one by one: single level and multi level wildcards.

Single Level: +

As the name already suggests, a single level wildcard is a substitute for one topic level. The plus symbol represents a single level wildcard in the topic.

topic_wildcard_plus

Any topic matches to a topic including the single level wildcard if it contains an arbitrary string instead of the wildcard. For example a subscription to myhome/groundfloor/+/temperature would match or not match the following topics:

topic_wildcard_plus_example

Multi Level: #

While the single level wildcard only covers one topic level, the multi level wildcard covers an arbitrary number of topic levels. In order to determine the matching topics it is required that the multi level wildcard is always the last character in the topic and it is preceded by a forward slash.

topic_wildcard_hash

topic_wildcard_hash_example

A client subscribing to a topic with a multi level wildcard is receiving all messages, which start with the pattern before the wildcard character, no matter how long or deep the topics will get. If you only specify the multilevel wildcard as a topic (#), it means that you will get every message sent over the MQTT broker. If you expect high throughput this is an anti pattern, see the best practices below.

Topics beginning with $

In general you are totally free in naming your topics, but there is one exception. Each topic, which starts with a $-symbol will be treated specially and is for example not part of the subscription when subscribing to #. These topics are reserved for internal statistics of the MQTT broker. Therefore it is not possible for clients to publish messages to these topics. At the moment there is no clear official standardization of topics that must be published by the broker. It is common practice to use $SYS/ for all these information and a lot of brokers implement these, but in different formats. One suggestion on $SYS-topics is in the MQTT GitHub wiki and here are some examples:

$SYS/broker/clients/connected
$SYS/broker/clients/disconnected
$SYS/broker/clients/total
$SYS/broker/messages/sent
$SYS/broker/uptime

Summary

So these were the basics about MQTT message topics. As you can see, MQTT topics are dynamically and give great flexibility to its creator. But when using these in real world applications there are some challenges you should be aware of. We collected our best practices, we learned the last year with excessively using MQTT in various projects. We are open to other suggestions or a discussion about these in the comments, so let us know your best practices or if you disagree with one of our best practices!

Best practices

Don’t use a leading forward slash

It is allowed to use a leading forward slash in MQTT, for example /myhome/groundfloor/livingroom. But that introduces a unnecessary topic level with a zero character at the front. That should be avoided, because it doesn’t provide any benefit and often leads to confusion.

Don’t use spaces in a topic

A space is the natural enemy of each programmer, they often make it much harder to read and debug topics, when things are not going the way, they should be. So similar to the first one, only because something is allowed doesn’t mean it should be used. UTF-8 knows many different white space types, it’s pretty obvious that such uncommon characters should be avoided.

Keep the topic short and concise

Each topic will be included in every message it is used in, so you should think about making them short and concise. When it comes to small devices, each byte counts and makes really a difference.

Use only ASCII characters, avoid non printable characters

Using non-ASCII UTF-8 character makes it really hard to find typos or issues related to the character set, because often they can not be displayed correctly. Unless it is really necessary we recommend avoid using non ASCII character in a topic.

Embed a unique identifier or the ClientId into the topic

In some cases it is very helpful, when the topic contains a unique identifier of the client the publish is coming from. This helps identifying, who send the message. Another advantage is the enforcement of authorization, so that only a client with the same ClientId as contained in the topic is allowed to publish to that topic. So a client with the id client1 is allowed to publish to client1/status, but not permitted to publish to client2/status.

Don’t subscribe to #

Sometimes it is necessary to subscribe to all messages, which are transferred over the broker, for example when persisting all of them into a database. This should not be done by using a MQTT client and subscribing to the multi level wildcard. The reason is that often the subscribing client is not able to process the load of messages that is coming its way. Especially if you have a massive throughput. Our recommended solution is to implement an extension in the MQTT broker, for example the plugin system of HiveMQ allows you to hook into the behavior of HiveMQ and add a asynchronous routine to process each incoming message and persist it to a database.

Don’t forget extensibility

Topics are a flexible concept and there is no need to preallocate them in any kind of way, regardless both the publisher and subscriber need to be aware of the topic. So it is important to think about how they can be extended in case you are adding new features to your product. For example when your smart home solution is extended by some new sensors, it should be possible to add these to your topic tree without changing the whole topic hierarchy.

Use specific topics, instead of general ones

When naming topics it is important not to use them like a queue, for example using only one topic for all messages is a anti pattern. You should use as specific topics as possible. So if you have three sensors in your living room, you should use topics myhome/livingroom/temperature, myhome/livingroom/brightness and myhome/livingroom/humidity, instead of sending all values over myhome/livingroom. Also this enables you to use other MQTT features like retained messages, which we cover in one of the next posts.


So that’s the end of part five in our MQTT Essentials series. We hope you enjoyed it. In the next post we cover the often mention Quality of Service (QoS) in MQTT. We’ll explain why this is an essential feature and how you can leverage it.

Have a great week and we’ll hope to see you on the next MQTT Monday!

If you still haven’t signed up for our newsletter in order to get each new post delivered directly to your inbox, you can do so below. If you prefer RSS, you can subscribe to our RSS feed here.

30 comments

  1. Ron Segal says:

    Great tutorial series.

    One comment on your best practices, i.e. ‘Sometimes it is necessary to subscribe to all messages, which are transferred over the broker, for example when persisting all of them into a database. This should not be done by using a MQTT client and subscribing to the multi level wildcard.’

    My initial thought was to implement something like this as a client that is physically local to the broker, which can then communicate with the database via a mechanism other than MQTT? Then if this still swamps a single local client, to segment the topics amongst multiple local clients. Then if this was to be done automatically it would require another client that listened only for published topics, allocating sub-sets to the local clients. Upshot is that although it probably could be done with such a scheme, for heavy loads a ‘plugin’ that entirely bypasses the client mechanism makes sense!

  2. Minh says:

    Publish a message to some independent topics.

    A client with the id client1 want to send a message to some clients with the ids: client 3, client 5, client7, and these destination clients do not subscribe to a same topic, how can MQTT handle this situation? ex: in a chat application, I want send a message to some friends without create a chat room, and these friends are independent

    1. Hi Minh,

      interesting question! Typically MQTT topics are designed by the application designer to handle this. A possible scenario would be that each client has some kind of “inbox” for such messages. So each client subscribes per default to a topic like:

      {myClientID}/inbox

      Of course client1 now needs to send the messages to “client3/inbox”, “client5/inbox” and “client7/inbox”. If that is not possible due to multiple messages, you can implement the message amplification with a plugin on the broker side, take a look at the PublishService (http://hivemq.com/docs/plugins/latest/#publish-service).

      A more elegant solution would be to use some kind of virtual rooms if possible where all of the clients subscribe to.

      Hope this helps,
      Dominik from the HiveMQ Team

  3. Paul says:

    I created a chat application which allows for P2p chat, Group Chat and Company broadcast message center. This uses STOMP instead of MQTT, but the context of the topic/queue is the same.
    For P2P, each user have a unique userid and when they login to the chat web app, they subscribe to their own queue/topic (aka inbox) queue/customerId/p2p/{MyUserId}, Th Chat Application know about all the users and their online/offline status via presence detection thru heartbeat (you can add a user to your favorite list), when Jane want to send a chat to John, she will click on John and push a message on John’s queue queue/customerId/p2p/1234 where John will thene receive it, the message has the userId of Jane , say 54321, when John responds he will post a message to queue/customerid/p2p/54321.
    for Group chat, the mechanism is the same but this time each logged in user knows of their group they belongs to so they get subscribed to a topic in the format topic/customerId/group/groupId. Now when a user belonging to the same group want to send a group message, they will publish the message to topic/customerId/group/55555 and all subscribers to topic/customerId/group/55555 will receive the message and can respond back. for company broadcast, the concept is still the same, every user subscribe to the company bulletin board, since it’s a corporate chat and supports multi-tennant each customer has their own unique customerId (bulletin board id per say) they will subscribe to topic/customerId/bulletin.

    Granted MQTT only does publish/subscribe (topic) and not queue (p2p), but you can do the same with pub/sub only. Hope this helps

  4. Steve Smith says:

    I am trying to figure if MQTT Client can be used on mutiple Topics in this format:

    Topic1: Client_1A,Client_1B etc
    Topic2 : Client_2A,Client_2B but also any client from Topic e.g. Client_1A etc

    Thing is that I might have different groups that can also contain clients from other groups. Hope that makes sense?

    SO when the client connects – it maybe connecting to multiple Topics at once.

    1. Hi Steve,

      Sure, a client can subscribe to multiple topics at once.

      Therefore see our post about MQTT Publish, Subscribe and Unsubscribe: http://www.hivemq.com/blog/mqtt-essentials-part-4-mqtt-publish-subscribe-unsubscribe

      Hope that helps,
      Christian from the HiveMQ Team

  5. Webber says:

    Nice tutorial , Really helps me a lot .

  6. ldsheng says:

    Nice! Really helpful and clear! thanks a lot.

  7. Sweetnsour says:

    Hi HiveMQ-Team, thanks for the cool site!

    I have (likely basic) questions about topic validation provided by standard MQTT…
    (Maybe that special $SYS/ topic will help here?)

    You say “There is no need for a client to create the desired topic before publishing or subscribing to it, because a broker accepts each valid topic without any prior initialization.” So I might have overread that, but how is a topic validated? I would think there must be some **control authority** at the broker where each valid topic would need to be registered on creation, and deleted on deletion. That instance would have the following tasks:

    2) So, if a client publishes to a topic “a”: client: publish(topic=”a”,payload=”ax”), the instance would **check whether this topic belongs to the application domain** at all. Otherwise that client could by mistake publish to topic “A”, and never reach its subscribers?

    3) Likewise, if another client subscribes to a topic “A”, the instance would perform the same check as above, meaning that if the topic is not registered (but “a” is), subscription should fail.

    4) That instance should be able to tell the current number of subscribers for a topic, and update in realtime, if subscriber number changes.

    5) Instance should be able to tell the current number of publishers for a topic, and update in realtime, if subscriber number changes. For example, if a topic has 0 subscribers, it could be considered for deletion.

    Does such an instance or control mechanism for valid topics of an application domain exist?

    Sunny regards, and thanks a lot in advance 🙂

    1. Sweetnsour says:

      Typo:
      5) The control instance should be able to tell the current number of publishers for a topic, and update in realtime, if *publisher* number changes. For example, if a topic has 0 subscribers or 0 publishers, it could be considered for deletion so as to avoid maybe millions of unused topic corpses.

    2. Hi Sweetnsour,

      an MQTT broker is not a message queue, so there are no predefined topics needed. Topics are of a dynamic nature in MQTT,

      regarding 2): this can happen when using MQTT it is the clients responsibility to provide the correct topic. With HiveMQ’s plugin system you can write a plugin which does that check if you need it, though.

      Hope that helps,
      the HiveMQ Team

  8. g says:

    very helpful thank you

  9. Graham Owens says:

    Hi

    Is it possible to create a symbolic link between topics in MQTT?

    For example, suppose that my phone was sending location updates to the following topic

    people/graham/location

    without publishing another topic, is there anything that can be done server side to link/mirror the specified topic into another topic, eg groups/family/graham/location so that a device could be built that subscribes to groups/family ?

    And if this is possible, is there any way to create/break the symbolic links from a client? This would then allow a device to subscribe to something home/people/+/location and have a MQTT client create (in this case my phone) create the symbolic link when I enter my home (derived from either home/location or home/wifi/ssid).

    If there is a structure that will allow the existing wildcards to work for these situations, please could you offer some advice.

    Best Regards

    1. Hi Graham,

      A symbolic link mechanism is not provided by the standard MQTT specification.

      But with HiveMQ’s plugin system it should be easy to write your own plugin which creates this behaviour.

      Hope that helps,
      Christoph from the HiveMQ Team

  10. vtvt says:

    What if some anonymous random user subscribes to my topic and see the contents of topic. What security features can be added for protection of topics and how?
    Also if pubclient1 created ‘topicx’; can pubclient2 create topic with same name ‘topicx’ in same server.
    Thanks in advance.

    1. We strongly recommend to enable authentication and authorization in order to prevent malicious clients for publishing and subscribing to arbitrary topics. We recommend to take a look at our MQTT Security Fundamentals Series which covers the topic in depth: http://www.hivemq.com/mqtt-security-fundamentals/

      Hope this helps,
      Dominik

  11. GYP says:

    I have 2 ESP client on same broker, each one subscribe different topic but in the same structure : ESP 1 (“livingroom/light/status”) and ESP 2 (“kitchen/light/status”), when both of them connected, system crash (I’m monitored from Serial Monitor), giving status looping on each Callback function and reconnect to wifi.
    Is my topic structure wrong? Any solution?
    Thanks

    1. Hallo,

      your topic structure is fine. Otherwise it’s hard to give any sort of advice. Nothing obvious springs to mind and it is not a known issue.

      Best regards,
      Florian, from the HiveMQ Team.

    2. GYP says:

      But, I cant connected both of them to same broker, it must be error. Actually, the complete topic for ESP 1 is “myhome/livingroom/light” and for command topic is “myhome/livingroom/light/status”, for ESP 2 the topic is “myhome/kitchen/light”, and the command is “myhome/kitchen/light/status”. So when ESP 1 & ESP 2 connect to same wifi, same broker, both of them like “crashing” topic with each other. So, can you give some advice for structure or topic naming? If, I changed one of them to “myhome/livingroom/light” for topic, and “myhome/livingroom/lightstatus” for command topic, they running well.

  12. GBird says:

    Hi
    Very helpful tutorial! … but I’m trying to publish using the topic ‘my_username@github/#’ (assigned by the broker) and got the error “publish topic cannot contain wildcards”. I would like to know what I’m doing wrong. Many thanks

    1. Hallo GBird,

      As the post above states.

      A wildcard can only be used when subscribing to topics and is not permitted when publishing a message.

      In your case you are trying to publish to a topic ending in the multi level wildcard chracter ‘#’.

      Kind regards,
      Florian, from The HiveMQ Team.

  13. aymen says:

    is there a way to use integer ranges in topic subscription and publishing like for example i want that clients are able to subscribe to a certain temperature sensor that are in a certain range
    home/sensor/[30..45]
    ?

    1. Hi,

      This is not possible. You will either need to create individual subscriptions for each of the temperatures or publish the data on a ‘range topic’, that you can then subscribe on.

      Best regards,
      Florian from The HiveMQ Team.

  14. KB says:

    If I subscribe to a topic let’s say myhome/# and a new topic is created myhome/kitchen after the subscription, will I receive the messages of that topic then?

    1. Hi there,

      your assumption is correct. Topics in MQTT are dynamic and a wildcard subscription will work for topics created after the subscription has happened.

      Kind regards,
      Florian From The HiveMQ Team.

  15. Niel Swart says:

    Hi

    Are all topics starting with ‘$’ reserved and therefore not adhering to the wildcard ‘rules’. I have an application where a device subscribes to the topic ${device.id}/something. If I would like to publish to all devices connected to the broker by using topic ‘+/something’ will that be possible?
    Many thanks

    1. Hi Niel,

      Glad to see you are taking an interest in MQTT and HiveMQ.
      Wildcards can only be used for Subscriptions, not for Publishes.

      Kind regards,
      Florian from The HiveMQ Team

Leave a Reply

Your email address will not be published. Required fields are marked *