MQTT Add-ons

HiveMQ 4.4 introduces MQTT Add-ons that extend the standard functionality of MQTT.

HiveMQ Topic Add-ons

HiveMQ provides special analytical MQTT topics that you can enable to automatically collect detailed information for all expired and dropped MQTT messages on your HiveMQ system. The new add-ons simplify access to all the data and metrics you need to analyze, report, or further process expired and dropped messages.

HiveMQ analytical topics use a reserved namespace that prevents conflicts with your existing workflows and applications. The reserved namespaces are denoted with $:

  • MQTT clients must subscribe to a topic on the $ namespace to receive data. For example, $expired/my-topic-of-interest/topic.

  • Only the HiveMQ broker can publish on a $ namespace, MQTT clients cannot publish messages on the reserved namespace.

  • $ namespaces are excluded from wildcard subscriptions.

Table 1. Available Analytical Topics
Topic Name Namespace Default Description

Expired Messages Topic

$expired

disabled

Captures all expired messages on the HiveMQ system.

Dropped Messages Topic

$dropped

disabled

Captures all dropped messages on the HiveMQ system.

Expired Messages Topic

The Expired Messages Topic add-on prefaces the topic of expired messages with $expired/.

Message expiry is a standard feature of the MQTT protocol that lets you define the length of time that can pass after a message arrives at your MQTT broker until the message expires. Messages that expire are never published. Message expiry can occur for several reasons:

  • A client takes too long to consume the message

  • The client remains offline and the message expires before the client can consume it

  • A retained message that is stored on the broker expires

For more information on message expiry and expiry configuration options, see Message Expiry.

Stored retained messages that expire are not redirected to the $expired topic.

Enable the Expired Messages Topic Add-on

  1. To enable the Expired Messages Topic add-on, go to your HiveMQ configuration XML file and locate the mqtt-addons section.

  2. Set the enabled tag of <expired-messages-topic> to true.

When the Expired Messages Topic add-on is enabled, HiveMQ publishes all queued messages and inflight messages that expire to $expired/<original topic>.

Users can utilize all existing MQTT and HiveMQ subscription and consumer mechanisms to interact with the expired messages.
For example, subscribe an MQTT client to $expired/# to get expired messages from all sources or configure HiveMQ enterprise extensions such as the Kafka extension to consume messages from $expired topics.

When you subscribe to an $expired topic, standard queuing mechanisms such as the maximum queue length and limits for shared subscriptions or consumers apply. We recommend the use of dedicated clients and shared subscriptions for analytical topics with appropriately configured queue limits.
Expired Messages Topic add-on configuration
<?xml version="1.0"?>
<hivemq xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    ...

    <mqtt-addons>
        <expired-messages-topic>
            <enabled>true</enabled>
        </expired-messages-topic>
    </mqtt-addons>
    ...
</hivemq>
HiveMQ publishes each expired message on $expired/<original-topic> as an MQTT 5 message and stores metadata from the original message in user properties. User properties are an MQTT 5 feature. MQTT 3 clients that subscribe to an $expired topic do not receive any of the information that is provided in the user properties of the $expired topic.
For more information, see Expired Messages Topic User Properties.

Expired Messages Topic User Properties

The Expired Messages Topic add-on publishes several additional user properties in each $expired/<original-topic> MQTT 5 message. The additional user properties appear after any user properties from the original message.

Message Expiry
Table 2. Available Expired Messages User Properties
User Property Name Description

hmq-exp-timestamp

The UNIX timestamp when the message expired

hmq-exp-origin

The origin type of the expired message

  • client

  • shared subscription

  • consumer

hmq-exp-client

When the origin type of the expired message is client, this property shows the client ID of the client for which the message expired

hmq-exp-shared-subscription

When the origin type of the expired message is shared subscription, this property shows the full shared subscription for which the queued/inflight message expired

hmq-exp-consumer

When the origin type of the expired message is consumer, this property shows the id of the consumer for which the message expired

Expired Messages Topic Available Information

The Expired Messages Topic add-on preserves the following information for each expired message:

To preserve the retained flag of the expired message, the subscription to the $expired topic must have a retain-as-published flag set.

Expired Messages Topic Metrics

The Expired Messages Topic add-on exposes additional metrics in the HiveMQ standard metrics:

Table 3. Available Expired Message Topic metrics
Metric Type Description

com.hivemq.messages.expired.cache-size

Gauge

The current size of the expired messages cache

com.hivemq.messages.expired.cache-full-misses.count

Counter

The number of times an expired message is not added to the expired message cache due to a full cache

com.hivemq.messages.expired.topic.dropped.count

Counter

The number of expired messages that are dropped due to queue limits

com.hivemq.messages.expired.topic.expired.count

Counter

The number of expired messages that expire in an $expired topic

com.hivemq.messages.expired.topic.enqueued.count

Counter

The number of expired messages that are queued to an $expired topic

Messages that expire within an $expired topic do not trigger publication of another expired message.

For a full list of HiveMQ metrics, see Available Metrics.

Dropped Messages Topic

The Dropped Messages Topic add-on prefaces the topic of dropped messages with $dropped/.

In MQTT, dropped messages are messages that the MQTT broker does not publish. Dropped messages can occur for various reasons.

For detailed information on dropped messages and the reasons messages are dropped, see Dropped Messages.

Enable the Dropped Messages Topic Add-on

  1. To enable the Dropped Messages Topic add-on, go to your HiveMQ configuration XML file and locate the mqtt-addons section.

  2. Set the enabled tag of <dropped-messages-topic> to true.

When the Dropped Messages Topic add-on is enabled, HiveMQ publishes all messages that are dropped to $dropped/<original topic>.

Users can utilize all existing MQTT and HiveMQ subscription and consumer mechanisms to interact with the dropped messages.
For example, subscribe an MQTT client to $dropped/# to get all dropped messages or configure HiveMQ enterprise extensions such as the Kafka extension to consume messages from a $dropped topic.

When you subscribe to a $dropped topic, standard queuing mechanisms such as the maximum queue length and limits for shared subscriptions or consumers apply. We recommend the use of dedicated clients and shared subscriptions for analytical topics with appropriately configured queue limits.

Dropped Messages Topic configuration
<?xml version="1.0"?>
<hivemq xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    ...

    <mqtt-addons>
        <dropped-messages-topic>
            <enabled>true</enabled>
        </dropped-messages-topic>
    </mqtt-addons>
    ...
</hivemq>
HiveMQ publishes each dropped message on $dropped/<original-topic> as an MQTT 5 message and stores metadata from the original message in user properties. User properties are an MQTT 5 feature. MQTT 3 clients that subscribe to the $dropped topic do not receive any of the information that is provided in the user properties of the $dropped topic. For more information, see Dropped Messages Topic User Properties.

Dropped Messages Topic User Properties

The Dropped Messages Topic add-on publishes several additional user properties in each $dropped/<original-topic> MQTT 5 message. The additional user properties appear after any user properties from the original message.

Dropped Message
Table 4. Available Dropped Messages User Properties
User Property Name Description

hmq-drop-timestamp

The UNIX timestamp when the broker dropped the message

hmq-drop-origin

The origin type of the dropped message. The following options are possible:

  • client

  • shared subscription

  • consumer

hmq-drop-client

When the origin type of the dropped message is client, this property shows the client ID of the client for which the queued/inflight message dropped

hmq-drop-shared-subscription

When the origin type of the dropped message is shared subscription, this property shows the full shared subscription for which the queued/inflight message dropped

NOTE: When a messages with QoS 0 is dropped for a shared subscription with the reason Internal Error or Not writable, both the client and shared subscription origin are present.

hmq-drop-consumer

When the origin type of the dropped message is consumer, this property shows the id of the consumer for which the queued/inflight message dropped

hmq-drop-reason

The reason why the message dropped. The following reasons are currently available:

  • Client Message Queue Full

  • QoS 0 Memory Exceeded

  • QoS 0 Channel Not Writable

  • Maximum Packet Size Exceeded

  • Extension Prevented (inbound)

  • Extension Prevented (outbound)

  • Internal Error

Dropped Messages Topic Available Information

The Dropped Messages Topic add-on preserves the following information for each dropped message:

To preserve the retained flag of the dropped message, the subscription to the $dropped topic must have a retain-as-published flag set.

Dropped Messages Topic Metrics

The Dropped Messages Topic add-on exposes additional metrics in the HiveMQ standard metrics:

Table 5. Available Dropped Message Topic metrics
Metric Type Description

com.hivemq.messages.dropped.topic.dropped.count

Counter

The number of dropped messages in a $dropped topic that are dropped due to queue limits

com.hivemq.messages.dropped.topic.message-created.client.count

Counter

The number of dropped messages in a $dropped topic that originate from clients

com.hivemq.messages.dropped.topic.message-created.shared.count

Counter

The number of dropped messages in a $dropped topic that originate from shared subscriptions

com.hivemq.messages.dropped.topic.message-created.consumer.count

Counter

The number dropped messages in a $dropped topic that originate from consumers

com.hivemq.messages.dropped.topic.enqueued.count

Counter

The number of dropped messages that are queued to a $dropped topic

For a full list of HiveMQ metrics, see Available Metrics.