HiveMQ MQTT Add-ons

HiveMQ MQTT Add-ons extend the standard functionality of MQTT to satisfy the needs of several key IoT uses cases.

HiveMQ Topic Add-ons

HiveMQ MQTT Topic Add-ons are special analytical MQTT topics that automatically collect detailed information for all expired and dropped MQTT messages on your HiveMQ system. The new HiveMQ Topic Add-ons simplify access to all the data and metrics you need to analyze, report, or further process your expired and dropped messages in external solutions such as Kafka. For more information, see Sample Use Cases.

HiveMQ analytical topics use a reserved namespace that prevents conflicts with your existing workflows and applications. The reserved namespaces are denoted with $:

  • MQTT clients must subscribe to a topic on the $ namespace to receive data. For example, $expired/my-topic-of-interest/topic.

  • Only the HiveMQ broker can publish on a $ namespace, MQTT clients cannot publish messages on the reserved namespace.

  • $ namespaces are excluded from wildcard subscriptions.

Table 1. Available Analytical Topics
Topic Name Namespace Default Description

Expired Messages Topic

$expired

disabled

Captures all expired messages on the HiveMQ system.

Dropped Messages Topic

$dropped

disabled

Captures all dropped messages on the HiveMQ system.

Expired Messages Topic

The Expired Messages Topic add-on prefaces the topic of expired messages with $expired/.

Message expiry is a standard feature of the MQTT protocol that lets you define the length of time that can pass after a message arrives at your MQTT broker until the message expires. Messages that expire are never published. Message expiry can occur for several reasons:

  • A client takes too long to consume the message

  • The client remains offline and the message expires before the client can consume it

  • A retained message that is stored on the broker expires

For more information on message expiry and expiry configuration options, see Message Expiry.

Stored retained messages that expire are not redirected to the $expired topic.

Enable the Expired Messages Topic Add-on

Enable the $expired topic to stay informed about all messages that expire on HiveMQ. The expired-message topic add-on provides useful data such as the client or shared subscription for which the message expired.

  1. To enable the Expired Messages Topic add-on, go to your HiveMQ configuration XML file and locate the mqtt-addons section.

  2. Set the enabled tag of <expired-messages-topic> to true.

When the Expired Messages Topic add-on is enabled, HiveMQ publishes all queued messages and inflight messages that expire to $expired/<original topic>.

You can utilize all existing MQTT and HiveMQ subscription and consumer mechanisms to interact with the expired messages.
For example, subscribe an MQTT client to $expired/# to get expired messages from all sources or configure HiveMQ enterprise extensions such as the Kafka extension to consume messages from $expired topics.

When you subscribe to an $expired topic, standard queuing mechanisms such as the maximum queue length and limits for shared subscriptions or consumers apply. We recommend the use of dedicated clients and shared subscriptions for analytical topics with appropriately configured queue limits.
Expired Messages Topic add-on configuration
<?xml version="1.0"?>
<hivemq xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    ...

    <mqtt-addons>
        <expired-messages-topic>
            <enabled>true</enabled>
        </expired-messages-topic>
    </mqtt-addons>
    ...
</hivemq>
HiveMQ publishes each expired message on $expired/<original-topic> as an MQTT 5 message and stores metadata from the original message in user properties. User properties are an MQTT 5 feature. MQTT 3 clients that subscribe to an $expired topic do not receive any of the information that is provided in the user properties of the $expired topic.
For more information, see Expired Messages Topic User Properties.

Expired Messages Topic User Properties

The Expired Messages Topic add-on publishes several additional user properties in each $expired/<original-topic> MQTT 5 message. The additional user properties appear after any user properties from the original message.

Message Expiry
Table 2. Available Expired Messages User Properties
User Property Name Description

hmq-exp-timestamp

The UNIX timestamp when the message expired

hmq-exp-origin

The origin type of the expired message

  • client

  • shared subscription

  • consumer

hmq-exp-client

When the origin type of the expired message is client, this property shows the client ID of the client for which the message expired

hmq-exp-shared-subscription

When the origin type of the expired message is shared subscription, this property shows the full shared subscription for which the queued/inflight message expired

hmq-exp-consumer

When the origin type of the expired message is consumer, this property shows the id of the consumer for which the message expired

Expired Messages Topic Available Information

The Expired Messages Topic add-on preserves the following information for each expired message:

To preserve the retained flag of the expired message, the subscription to the $expired topic must have a retain-as-published flag set.

Expired Messages Topic Metrics

The Expired Messages Topic add-on exposes additional metrics in the HiveMQ standard metrics:

Table 3. Available Expired Message Topic metrics
Metric Type Description

com.hivemq.messages.expired.cache-size

Gauge

The current size of the expired messages cache

com.hivemq.messages.expired.cache-full-misses.count

Counter

The number of times an expired message is not added to the expired message cache due to a full cache

com.hivemq.messages.expired.topic.dropped.count

Counter

The number of expired messages that are dropped due to queue limits

com.hivemq.messages.expired.topic.expired.count

Counter

The number of expired messages that expire in an $expired topic

com.hivemq.messages.expired.topic.enqueued.count

Counter

The number of expired messages that are queued to an $expired topic

Messages that expire within an $expired topic do not trigger publication of another expired message.

For a full list of HiveMQ metrics, see Available Metrics.

Dropped Messages Topic

The Dropped Messages Topic add-on prefaces the topic of dropped messages with $dropped/.

In MQTT, dropped messages are messages that the MQTT broker does not publish. Dropped messages can occur for various reasons.

For detailed information on dropped messages and the reasons messages are dropped, see Dropped Messages.

Enable the Dropped Messages Topic Add-on

Enable the $dropped topic to gain an overview of all dropped messages on your HiveMQ system. The dropped-messages topic includes key information such as the dropped reason code and the client for which the message was dropped.

  1. To enable the Dropped Messages Topic add-on, go to your HiveMQ configuration XML file and locate the mqtt-addons section.

  2. Set the enabled tag of <dropped-messages-topic> to true.

When the Dropped Messages Topic add-on is enabled, HiveMQ publishes all messages that are dropped to $dropped/<original topic>.

You can utilize all existing MQTT and HiveMQ subscription and consumer mechanisms to interact with the dropped messages.
For example, subscribe an MQTT client to $dropped/# to get all dropped messages or configure HiveMQ enterprise extensions such as the Kafka extension to consume messages from a $dropped topic.

When you subscribe to a $dropped topic, standard queuing mechanisms such as the maximum queue length and limits for shared subscriptions or consumers apply. We recommend the use of dedicated clients and shared subscriptions for analytical topics with appropriately configured queue limits.

Dropped Messages Topic configuration
<?xml version="1.0"?>
<hivemq xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    ...

    <mqtt-addons>
        <dropped-messages-topic>
            <enabled>true</enabled>
        </dropped-messages-topic>
    </mqtt-addons>
    ...
</hivemq>
HiveMQ publishes each dropped message on $dropped/<original-topic> as an MQTT 5 message and stores metadata from the original message in user properties. User properties are an MQTT 5 feature. MQTT 3 clients that subscribe to the $dropped topic do not receive any of the information that is provided in the user properties of the $dropped topic. For more information, see Dropped Messages Topic User Properties.

Dropped Messages Topic User Properties

The Dropped Messages Topic add-on publishes several additional user properties in each $dropped/<original-topic> MQTT 5 message. The additional user properties appear after any user properties from the original message.

Dropped Message
Table 4. Available Dropped Messages User Properties
User Property Name Description

hmq-drop-timestamp

The UNIX timestamp when the broker dropped the message

hmq-drop-origin

The origin type of the dropped message. The following options are possible:

  • client

  • shared subscription

  • consumer

hmq-drop-client

When the origin type of the dropped message is client, this property shows the client ID of the client for which the queued/inflight message dropped

hmq-drop-shared-subscription

When the origin type of the dropped message is shared subscription, this property shows the full shared subscription for which the queued/inflight message dropped

NOTE: When a messages with QoS 0 is dropped for a shared subscription with the reason Internal Error or Not writable, both the client and shared subscription origin are present.

hmq-drop-consumer

When the origin type of the dropped message is consumer, this property shows the id of the consumer for which the queued/inflight message dropped

hmq-drop-reason

The reason why the message dropped. The following reasons are currently available:

  • Client Message Queue Full

  • QoS 0 Memory Exceeded

  • QoS 0 Channel Not Writable

  • Maximum Packet Size Exceeded

  • Extension Prevented (inbound)

  • Extension Prevented (outbound)

  • Internal Error

Dropped Messages Topic Available Information

The Dropped Messages Topic add-on preserves the following information for each dropped message:

To preserve the retained flag of the dropped message, the subscription to the $dropped topic must have a retain-as-published flag set.

Dropped Messages Topic Metrics

The Dropped Messages Topic add-on exposes additional metrics in the HiveMQ standard metrics:

Table 5. Available Dropped Message Topic metrics
Metric Type Description

com.hivemq.messages.dropped.topic.dropped.count

Counter

The number of dropped messages in a $dropped topic that are dropped due to queue limits

com.hivemq.messages.dropped.topic.message-created.client.count

Counter

The number of dropped messages in a $dropped topic that originate from clients

com.hivemq.messages.dropped.topic.message-created.shared.count

Counter

The number of dropped messages in a $dropped topic that originate from shared subscriptions

com.hivemq.messages.dropped.topic.message-created.consumer.count

Counter

The number dropped messages in a $dropped topic that originate from consumers

com.hivemq.messages.dropped.topic.enqueued.count

Counter

The number of dropped messages that are queued to a $dropped topic

For a full list of HiveMQ metrics, see Available Metrics.

Use Cases

Messages that expire or drop from your MQTT broker are never delivered to clients. In some instances, lack of delivery is desirable. In other cases, message expiration or dropped messages requires further attention. Previously, expired and dropped message data was limited and difficult to obtain. HiveMQ developed MQTT Topic Add-ons to simplify access to all the expired and dropped message data and metrics you need to understand and optimize your IoT applications.

Expired Messages Topic: Smart Car Use Case

In this example, a smart-car manufacturer offers a mobile app that customers use to unlock specific smart cars. To ensure customer satisfaction and fine-tune app functionality, the smart-car manufacturer requires precise information about app performance. Upon request, the mobile app sends an MQTT message to unlock the door of a smart car. If the requested MQTT message fails to arrive at the car within a predefined number of seconds, the message expires on the HiveMQ broker. In this case, message expiry is crucial since late delivery of the MQTT message could cause a car door to open at an inappropriate moment.

To gather detailed information on how often messages fail to successfully open a car door within the designated period, the smart-car manufacturere implements the HiveMQ Expired Messages Topic MQTT Add-on. Once the feature is enabled, HiveMQ automatically stores all expired messages to a special MQTT analytical topic. The car manufacturer uses the open-source MQTT CLI (optional) and the HiveMQ Enterprise Extension for Kafka to forward the expired message data to their Kafka cluster for further analysis. For example, to identify which car models are frequently affected.

Example subscribe to the $expired MQTT topic with the MQTT CLI
mqtt sub -t $expired/# -h your.broker
Example forward expired message data to Kafka with the HiveMQ Kafka extension
    <mqtt-to-kafka-mapping>
        <id>expired-topic-mapping</id>

        <!-- Kafka cluster to use for this topic mapping -->
        <cluster-id>cluster01</cluster-id>

        <!-- List of MQTT topic filters -->
        <mqtt-topic-filters>
            <mqtt-topic-filter>$expired/#</mqtt-topic-filter>
        </mqtt-topic-filters>

        <!-- Target Kafka topic -->
        <kafka-topic>expired-kafka-topic</kafka-topic>

    </mqtt-to-kafka-mapping>

Dropped Messages Topic: Fleet Management Use Case

Dropped messages can occur for various reasons and are seldom the desired outcome. In this example, a fleet management company implements the HiveMQ Dropped Messages Topic Add-on to ensure the integrity of their legal reporting. A trucking company uses MQTT messages to track driving times for each driver in the fleet. To comply with stringent legal requirements, it is imperative that all records are accurate and complete. The fleet management company relies on the HiveMQ Dropped Messages Topic Add-on to automatically store all dropped messages. Although the messages were not delivered to the desired recipient, no data is permanently lost.

To preserve a complete history and analyze the data further, the fleet management company uses the MQTT CLI (optional) and the HiveMQ Enterprose Extansion for Kafka to forward the accumulated dropped message data to their Kafka cluster.

Example subscribe to the $dropped MQTT topic with the MQTT CLI
mqtt sub -t $dropped/# -h your.broker
Example forward dropped message data to Kafka with the HiveMQ Kafka extension
    <mqtt-to-kafka-mapping>
        <id>dropped-topic-mapping</id>

        <!-- Kafka cluster to use for this topic mapping -->
        <cluster-id>cluster01</cluster-id>

        <!-- List of MQTT topic filters -->
        <mqtt-topic-filters>
            <mqtt-topic-filter>$dropped/#</mqtt-topic-filter>
        </mqtt-topic-filters>

        <!-- Target Kafka topic -->
        <kafka-topic>dropped-kafka-topic</kafka-topic>

    </mqtt-to-kafka-mapping>