MQTT Essentials Part 5: MQTT Topics & Best Practices

mqttessentials_part5

Welcome to the fifth part of MQTT Essentials. A ten-part blog series on the core features and concepts of the MQTT protocol. In this post, we focus on MQTT topics and best practices. As we have already mentioned, the MQTT broker uses the topic of a message to decide which client receives which message. We also look at SYS-topics, which are special topics that reveal information about the broker itself.

Let’s get started.

Topics

In MQTT, the word topic refers to an UTF-8 string that the broker uses to filter messages for each connected client. The topic consists of one or more topic levels. Each topic level is separated by a forward slash (topic level separator).

topic_basics

In comparison to a message queue, MQTT topics are very lightweight. The client does not need to create the desired topic before they publish or subscribe to it. The broker accepts each valid topic without any prior initialization.

Here are some examples of topics:

myhome/groundfloor/livingroom/temperature
USA/California/San Francisco/Silicon Valley
5ff4a2ce-e485-40f4-826c-b1a5d81be9b6/status
Germany/Bavaria/car/2382340923453/latitude

Note that each topic must contain at least 1 character and that the topic string permits empty spaces. Topics are case-sensitive. For example, myhome/temperature and MyHome/Temperature are two different topics. Additionally, the forward slash alone is a valid topic.

Wildcards

When a client subscribes to a topic, it can subscribe to the exact topic of a published message or it can use wildcards to subscribe to multiple topics simultaneously. A wildcard can only be used to subscribe to topics, not to publish a message. There are two different kinds of wildcards: single-level and multi-level.

Single Level: +

As the name suggests, a single-level wildcard replaces one topic level. The plus symbol represents a single-level wildcard in a topic.

topic_wildcard_plus

Any topic matches a topic with single-level wildcard if it contains an arbitrary string instead of the wildcard. For example a subscription to myhome/groundfloor/+/temperature can produce the following results:

topic_wildcard_plus_example

Multi Level: #

The multi-level wildcard covers many topic levels. The hash symbol represents the multi-level wild card in the topic. For the broker to determine which topics match, the multi-level wildcard must be placed as the last character in the topic and preceded by a forward slash.

topic_wildcard_hash

topic_wildcard_hash_example

When a client subscribes to a topic with a multi-level wildcard, it receives all messages of a topic that begins with the pattern before the wildcard character, no matter how long or deep the topic is. If you specify only the multi-level wildcard as a topic (#), you receive all messages that are sent to the MQTT broker. If you expect high throughput, subscription with a multi-level wildcard alone is an anti-pattern (see the best practices below).

Topics beginning with $

Generally, you can name your MQTT topics as you wish. However, there is one exception: Topics that start with a $ symbol have a different purpose. These topics are not part of the subscription when you subscribe to the multi-level wildcard as a topic (#). The $-symbol topics are reserved for internal statistics of the MQTT broker. Clients cannot publish messages to these topics. At the moment, there is no official standardization for such topics. Commonly, $SYS/ is used for all the following information, but broker implementations varies. One suggestion for $SYS-topics is in the MQTT GitHub wiki. Here are some examples:

$SYS/broker/clients/connected
$SYS/broker/clients/disconnected
$SYS/broker/clients/total
$SYS/broker/messages/sent
$SYS/broker/uptime

Summary

These are the basics of MQTT message topics. As you can see, MQTT topics are dynamic and provide great flexibility. When you use wildcards in real-world applications, there are some challenges you should be aware of. We have collected the best practices that we have learned from working extensively with MQTT in various projects and are always open to suggestions or a discussion about these practices. Use the comments to start a conversation, Let us know your best practices or if you disagree with one of ours!

Best practices

Never use a leading forward slash

A leading forward slash is permitted in MQTT. For example, /myhome/groundfloor/livingroom. However, the leading forward slash introduces an unnecessary topic level with a zero character at the front. The zero does not provide any benefit and often leads to confusion.

Never use spaces in a topic

A space is the natural enemy of every programmer. When things are not going the way they should, spaces make it much harder to read and debug topics. As with leading forward slashes, just because something is allowed, doesn’t mean it should be used. UTF-8 has many different white space types, such uncommon characters should be avoided.

Keep the topic short and concise

Each topic is included in every message in which it is used. Make your topics as short and concise as possible. When it comes to small devices, every byte counts and topic length has a big impact.

Use only ASCII characters, avoid non printable characters

Because non-ASCII UTF-8 characters often display incorrectly, it is very difficult to find typos or issues related to the character set. Unless it is absolutely necessary, we recommend avoiding the use of non-ASCII characters in a topic.

Embed a unique identifier or the Client Id into the topic

It can be very helpful to include the unique identifier of the publishing client in the topic. The unique identifier in the topic helps you identify who sent the message. The embedded ID can be used to enforce authorization. Only a client that has the same client ID as the ID in the topic is allowed to publish to that topic. For example, a client with the client1 ID is allowed to publish to client1/status, but not permitted to publish to client2/status.

Don’t subscribe to #

Sometimes, it is necessary to subscribe to all messages that are transferred over the broker. For example, to persist all messages into a database. Do not subscribe to all messages on a broker by using an MQTT client and subscribing to a multi-level wildcard. Frequently, the subscribing client is not able to process the load of messages that results from this method (especially if you have a massive throughput). Our recommendation is to implement an extension in the MQTT broker. For example, with the plugin system of HiveMQ you can hook into the behavior of HiveMQ and add an asynchronous routine to process each incoming message and persist it to a database.

Don’t forget extensibility

Topics are a flexible concept and there is no need to preallocate them in any way. However, both the publisher and the subscriber need to be aware of the topic. It is important to think about how topics can be extended to allow for new features or products. For example, if your smart-home solution adds new sensors, it should be possible to add these to your topic tree without changing the whole topic hierarchy.

Use specific topics, not general ones

When you name topics, don’t use them in the same way as in a queue. Be as specific topics as possible. For example, if you have three sensors in your living room, create topics for myhome/livingroom/temperature, myhome/livingroom/brightness and myhome/livingroom/humidity. Do not send all values over myhome/livingroom. Use of a single topic for all messages is a anti pattern. Specific naming also makes it possible for you to use other MQTT features such as retained messages. For more on retained messages, see part 8 of the Essentials series.


That brings us to the end of part five in our MQTT Essentials series. Thanks for reading. In the next post, we cover Quality of Service (QoS) in MQTT. We’ll explain why this is an essential feature and how you can leverage it.

Have a great week and we look forward to seeing you for the next MQTT Monday!

If you would like to have our newsletter delivered directly to your inbox as soon as it is released, enter your email in the subscription form below. If you prefer RSS, you can subscribe to our RSS feed here.

50 comments

  1. Ron Segal says:

    Great tutorial series.

    One comment on your best practices, i.e. ‘Sometimes it is necessary to subscribe to all messages, which are transferred over the broker, for example when persisting all of them into a database. This should not be done by using a MQTT client and subscribing to the multi level wildcard.’

    My initial thought was to implement something like this as a client that is physically local to the broker, which can then communicate with the database via a mechanism other than MQTT? Then if this still swamps a single local client, to segment the topics amongst multiple local clients. Then if this was to be done automatically it would require another client that listened only for published topics, allocating sub-sets to the local clients. Upshot is that although it probably could be done with such a scheme, for heavy loads a ‘plugin’ that entirely bypasses the client mechanism makes sense!

  2. Minh says:

    Publish a message to some independent topics.

    A client with the id client1 want to send a message to some clients with the ids: client 3, client 5, client7, and these destination clients do not subscribe to a same topic, how can MQTT handle this situation? ex: in a chat application, I want send a message to some friends without create a chat room, and these friends are independent

    1. Hi Minh,

      interesting question! Typically MQTT topics are designed by the application designer to handle this. A possible scenario would be that each client has some kind of “inbox” for such messages. So each client subscribes per default to a topic like:

      {myClientID}/inbox

      Of course client1 now needs to send the messages to “client3/inbox”, “client5/inbox” and “client7/inbox”. If that is not possible due to multiple messages, you can implement the message amplification with a plugin on the broker side, take a look at the PublishService (http://hivemq.com/docs/plugins/latest/#publish-service).

      A more elegant solution would be to use some kind of virtual rooms if possible where all of the clients subscribe to.

      Hope this helps,
      Dominik from the HiveMQ Team

  3. Paul says:

    I created a chat application which allows for P2p chat, Group Chat and Company broadcast message center. This uses STOMP instead of MQTT, but the context of the topic/queue is the same.
    For P2P, each user have a unique userid and when they login to the chat web app, they subscribe to their own queue/topic (aka inbox) queue/customerId/p2p/{MyUserId}, Th Chat Application know about all the users and their online/offline status via presence detection thru heartbeat (you can add a user to your favorite list), when Jane want to send a chat to John, she will click on John and push a message on John’s queue queue/customerId/p2p/1234 where John will thene receive it, the message has the userId of Jane , say 54321, when John responds he will post a message to queue/customerid/p2p/54321.
    for Group chat, the mechanism is the same but this time each logged in user knows of their group they belongs to so they get subscribed to a topic in the format topic/customerId/group/groupId. Now when a user belonging to the same group want to send a group message, they will publish the message to topic/customerId/group/55555 and all subscribers to topic/customerId/group/55555 will receive the message and can respond back. for company broadcast, the concept is still the same, every user subscribe to the company bulletin board, since it’s a corporate chat and supports multi-tennant each customer has their own unique customerId (bulletin board id per say) they will subscribe to topic/customerId/bulletin.

    Granted MQTT only does publish/subscribe (topic) and not queue (p2p), but you can do the same with pub/sub only. Hope this helps

  4. Steve Smith says:

    I am trying to figure if MQTT Client can be used on mutiple Topics in this format:

    Topic1: Client_1A,Client_1B etc
    Topic2 : Client_2A,Client_2B but also any client from Topic e.g. Client_1A etc

    Thing is that I might have different groups that can also contain clients from other groups. Hope that makes sense?

    SO when the client connects – it maybe connecting to multiple Topics at once.

    1. Hi Steve,

      Sure, a client can subscribe to multiple topics at once.

      Therefore see our post about MQTT Publish, Subscribe and Unsubscribe: http://www.hivemq.com/blog/mqtt-essentials-part-4-mqtt-publish-subscribe-unsubscribe

      Hope that helps,
      Christian from the HiveMQ Team

  5. Webber says:

    Nice tutorial , Really helps me a lot .

  6. ldsheng says:

    Nice! Really helpful and clear! thanks a lot.

  7. Sweetnsour says:

    Hi HiveMQ-Team, thanks for the cool site!

    I have (likely basic) questions about topic validation provided by standard MQTT…
    (Maybe that special $SYS/ topic will help here?)

    You say “There is no need for a client to create the desired topic before publishing or subscribing to it, because a broker accepts each valid topic without any prior initialization.” So I might have overread that, but how is a topic validated? I would think there must be some **control authority** at the broker where each valid topic would need to be registered on creation, and deleted on deletion. That instance would have the following tasks:

    2) So, if a client publishes to a topic “a”: client: publish(topic=”a”,payload=”ax”), the instance would **check whether this topic belongs to the application domain** at all. Otherwise that client could by mistake publish to topic “A”, and never reach its subscribers?

    3) Likewise, if another client subscribes to a topic “A”, the instance would perform the same check as above, meaning that if the topic is not registered (but “a” is), subscription should fail.

    4) That instance should be able to tell the current number of subscribers for a topic, and update in realtime, if subscriber number changes.

    5) Instance should be able to tell the current number of publishers for a topic, and update in realtime, if subscriber number changes. For example, if a topic has 0 subscribers, it could be considered for deletion.

    Does such an instance or control mechanism for valid topics of an application domain exist?

    Sunny regards, and thanks a lot in advance 🙂

    1. Sweetnsour says:

      Typo:
      5) The control instance should be able to tell the current number of publishers for a topic, and update in realtime, if *publisher* number changes. For example, if a topic has 0 subscribers or 0 publishers, it could be considered for deletion so as to avoid maybe millions of unused topic corpses.

    2. Hi Sweetnsour,

      an MQTT broker is not a message queue, so there are no predefined topics needed. Topics are of a dynamic nature in MQTT,

      regarding 2): this can happen when using MQTT it is the clients responsibility to provide the correct topic. With HiveMQ’s plugin system you can write a plugin which does that check if you need it, though.

      Hope that helps,
      the HiveMQ Team

  8. g says:

    very helpful thank you

  9. Graham Owens says:

    Hi

    Is it possible to create a symbolic link between topics in MQTT?

    For example, suppose that my phone was sending location updates to the following topic

    people/graham/location

    without publishing another topic, is there anything that can be done server side to link/mirror the specified topic into another topic, eg groups/family/graham/location so that a device could be built that subscribes to groups/family ?

    And if this is possible, is there any way to create/break the symbolic links from a client? This would then allow a device to subscribe to something home/people/+/location and have a MQTT client create (in this case my phone) create the symbolic link when I enter my home (derived from either home/location or home/wifi/ssid).

    If there is a structure that will allow the existing wildcards to work for these situations, please could you offer some advice.

    Best Regards

    1. Hi Graham,

      A symbolic link mechanism is not provided by the standard MQTT specification.

      But with HiveMQ’s plugin system it should be easy to write your own plugin which creates this behaviour.

      Hope that helps,
      Christoph from the HiveMQ Team

  10. vtvt says:

    What if some anonymous random user subscribes to my topic and see the contents of topic. What security features can be added for protection of topics and how?
    Also if pubclient1 created ‘topicx’; can pubclient2 create topic with same name ‘topicx’ in same server.
    Thanks in advance.

    1. We strongly recommend to enable authentication and authorization in order to prevent malicious clients for publishing and subscribing to arbitrary topics. We recommend to take a look at our MQTT Security Fundamentals Series which covers the topic in depth: http://www.hivemq.com/mqtt-security-fundamentals/

      Hope this helps,
      Dominik

  11. GYP says:

    I have 2 ESP client on same broker, each one subscribe different topic but in the same structure : ESP 1 (“livingroom/light/status”) and ESP 2 (“kitchen/light/status”), when both of them connected, system crash (I’m monitored from Serial Monitor), giving status looping on each Callback function and reconnect to wifi.
    Is my topic structure wrong? Any solution?
    Thanks

    1. Hallo,

      your topic structure is fine. Otherwise it’s hard to give any sort of advice. Nothing obvious springs to mind and it is not a known issue.

      Best regards,
      Florian, from the HiveMQ Team.

    2. GYP says:

      But, I cant connected both of them to same broker, it must be error. Actually, the complete topic for ESP 1 is “myhome/livingroom/light” and for command topic is “myhome/livingroom/light/status”, for ESP 2 the topic is “myhome/kitchen/light”, and the command is “myhome/kitchen/light/status”. So when ESP 1 & ESP 2 connect to same wifi, same broker, both of them like “crashing” topic with each other. So, can you give some advice for structure or topic naming? If, I changed one of them to “myhome/livingroom/light” for topic, and “myhome/livingroom/lightstatus” for command topic, they running well.

  12. GBird says:

    Hi
    Very helpful tutorial! … but I’m trying to publish using the topic ‘my_username@github/#’ (assigned by the broker) and got the error “publish topic cannot contain wildcards”. I would like to know what I’m doing wrong. Many thanks

    1. Hallo GBird,

      As the post above states.

      A wildcard can only be used when subscribing to topics and is not permitted when publishing a message.

      In your case you are trying to publish to a topic ending in the multi level wildcard chracter ‘#’.

      Kind regards,
      Florian, from The HiveMQ Team.

  13. aymen says:

    is there a way to use integer ranges in topic subscription and publishing like for example i want that clients are able to subscribe to a certain temperature sensor that are in a certain range
    home/sensor/[30..45]
    ?

    1. Hi,

      This is not possible. You will either need to create individual subscriptions for each of the temperatures or publish the data on a ‘range topic’, that you can then subscribe on.

      Best regards,
      Florian from The HiveMQ Team.

  14. KB says:

    If I subscribe to a topic let’s say myhome/# and a new topic is created myhome/kitchen after the subscription, will I receive the messages of that topic then?

    1. Hi there,

      your assumption is correct. Topics in MQTT are dynamic and a wildcard subscription will work for topics created after the subscription has happened.

      Kind regards,
      Florian From The HiveMQ Team.

  15. Niel Swart says:

    Hi

    Are all topics starting with ‘$’ reserved and therefore not adhering to the wildcard ‘rules’. I have an application where a device subscribes to the topic ${device.id}/something. If I would like to publish to all devices connected to the broker by using topic ‘+/something’ will that be possible?
    Many thanks

    1. Hi Niel,

      Glad to see you are taking an interest in MQTT and HiveMQ.
      Wildcards can only be used for Subscriptions, not for Publishes.

      Kind regards,
      Florian from The HiveMQ Team

    2. Dip says:

      Hi Niel,

      In your case, your every devices has to subscribe for “+/something” then every messages either “${device.id}/something” or “Something” would be publish for all of your devices. For an example – If some sensor publish something on specific Topic “${device1}/something” then broker will definitely publish to device1 but it will also publish to other device those already subscribed for “+/something”. At the end your every devices will receive message with topic “+/something”.

      Hope that will help you.

      Best Regards,
      Dip

    3. Hi Niel,

      Nice to see you’re taking an interest in MQTT and HiveMQ.

      A subscribe to “+/something” will not subscribe to any SYS topics. In order to get the publishes from the SYS topics, you need to explicitly subscribe with “$SYS/something”.

      Kind regards,
      Abdullah from the HiveMQ Team.

  16. Serkan Urkmez says:

    Which will be faster method for QoS 0, no ACK?
    I want to send ramdomly ON or OFF message to 1000 clients at once.
    This is time critical very under second.

    Method 1:
    Publish (Topic/Message): 1/ON, 2/OFF, 3/OFF, 4/ON, 5/OFF, ……….(Subscriptions for numbers)

    Method 2:
    Publish (MainTopic/Message JSON): MainTopic/{1: ON, 2: OFF, 3:OFF, 4:ON, 5:OFF, ………} (Subscription for MainTopic only)

    I don’t know about TCP or if there is async communication level.

    Thank you

    1. Hallo Serkan,

      Nice to see you are taking an interest in MQTT and HiveMQ.
      Without knowing any more details about your use case my suggestion would be that method 2 is faster, as you have less overhead form the TCP and MQTT protocol and there is also less network delay.

      Kind regards,
      Florian

  17. Long says:

    Thank you, HiveMQ Team.

    Your article is very helpful. Very clear and easy to understand.

    BTW, I have a question regarding to topic design.

    As I see, there’re 3 commons targets in IOT world:
    1. Manage devices: add, remove.
    2. Get devices status.
    3. Control devices.

    Could you please tell me why we should create topic under this format: home/room/dev_type/dev_id ?
    I’m sorry, I could not see any benefit to do that. And also don’t know how to use it.

    From my point of view, if we just have 2 kind of topic:
    – clients/dev/dev_id/stat
    – clients/dev/dev_id/ctrl

    It’s easier to:
    – Add a new device as well as remove it.
    – Send control message to specified device.

    Thank you very much,

    1. Hi Long,

      Nice to see you’re taking an interest in MQTT and HiveMQ and thank you for the kind words.
      The possibilities of MQTT and how they can be integrated into real life use cases are manifold.
      Each specific use case, dependant on the surrounding eco system and a number of possible restrictions or requirements is unique and can therefore call for a unique topic structure.
      Additionally personal preference does also play a role.
      How you structure your topics is completely up to you and if you believe that your proposal makes it easier to use then you should certainly go for it.

      I hope this answers your question.
      Kind regards,
      Florian from The HiveMQ Team.

    2. Long says:

      Thanks for your response.
      I am understand what you mean. I just wonder what professional will do

      The fact is I am very new, an idiot 😀
      I am not sure the way I am doing is the good way.

      If you could share some experience related, please. It also helps others.

      Thanks & Best regards,
      Long

    3. Hi Long,

      The way you design your topic structure is up to you.
      When your approach helps you implement your use case everything is fine. There is no “right professional approach”.

      Kind regards,
      Florian form The HiveMQ Team

  18. Omid Aghakhani says:

    Hi
    I think to can we have hook for MQTT :think
    Example for each message that broker recieve,run hook and send message to it for argument
    Or other thing
    Can i add hook on MQTT?

    Top Regard 🙂

    1. Hi Omit,

      Nice to see you’re taking an interest in MQTT.
      The HiveMQ Plugin System offers a variety of callbacks that can be used as hooks to extend the MQTT message flow with any sort of custom logic.

      Kind regards,
      Florian from the HiveMQ Team.

  19. Salvatore Coppola says:

    Great tutorial!

    1. Hi Salvatore,

      thank you for the kind words.

      Kind regards,
      Abdullah from the HiveMQ Team

  20. Omid Aghakhani says:

    Hi
    Thank you for the your replies
    With your good tutorial i could implement MQTT broker and use in my project
    I’m needing know that how many connection mqtt can accept?

    1. Hi Omid,

      it’s nice to hear that we could contribute to your project.

      In regards to your question, there is no limit for MQTT or HiveMQ defined. The only limitation would be the available resources.

      Kind regards,
      Abdullah from the HiveMQ Team.

  21. Davide Icardi says:

    I’m working for a project where devices push to topics defined like “deviceID1/xyz”, “deviceID2/xyz”, “deviceID3/xyz”.

    It is possible to distribute the messages between subscribers based on device id? For example subscriber 1 subscribe to “deviceID1 to deviceID100”, subscriber 2 to “deviceID101 to deviceID200”? Maybe using a regex or something similar?

    If this is not possible what do you suggest to distribute traffic between two or more servers? Shared subscribtions?

    1. Hi Davide,

      nice to see you’re taking an interest in MQTT and HiveMQ.

      Personally the right approach for this would be to use shared subscription.
      For that the subscriber can subscribe to “$share/devices/+/xyz”.

      This would distribute all incoming messages to “deviceID1/xyz”, “deviceID2/xyz” and “deviceID3/xyz” between the subscriber.

      Kind regards,
      Abdullah from the HiveMQ Team.

  22. Davide Icardi says:

    Is ordering of messages guaranteed when using Shared Subscriptions? If not there are some ways to guarantee ordering and scale the consumers?

    thanks

    1. Hi Davide,

      nice to see you’re taking an interest in MQTT and HiveMQ.

      The ordering of messages in shared subscriptions is guaranteed when QoS 1 is used.

      Kind regards,
      Abdullah from the HiveMQ Team.

  23. Ayomide Olalandu says:

    Is it possible to have multiple single level wildcards.
    An Example:

    topics – >
    1. parent/+/+/child
    2. parent/+/middleman/+/child
    3. parent/+/+
    4. +/+/child

    Thanks in advance!

    1. Hello Ayomide,

      Great to see you’re taking an interest in MQTT and HiveMQ.
      Short answer: Yes! MQTT allows for multiple single level wildcard in an individual subscription.
      All of your examples are valid.

      Hope this helps.
      Kind regards,
      Florian from the HiveMQ Team.

Leave a Reply

Your email address will not be published. Required fields are marked *