Monitoring

System monitoring is an essential part of every production-software deployment. Monitoring your MQTT brokers is vital, especially in clustered environments. HiveMQ is designed to help you enable different kinds of monitoring easily. When you run HiveMQ in critical infrastructure, we strongly recommend use of an appropriate monitoring application.

The highly-performant HiveMQ metrics subsystem allows you to monitor relevant metrics in low-latency high-throughput environments with no reduction in system performance.

JMX

HiveMQ provides extensive support for Java Management Extensions (JMX) to monitor internals of HiveMQ and the Java virtual machine (JVM). JMX is a proven industry standard for Java monitoring and many external tools support JMX natively or via extensions. The HiveMQ core distribution ships with the HiveMQ JVM Metrics Plugin and the HiveMQ JMX Plugin. If you prefer not to use JMX for monitoring, simply delete the JMX plugin from your plugin directory.

Be sure to set all JMX options properly. For more information, see JMX documentation.

The JVM Metrics Plugin adds several additional metrics about the JVM on which HiveMQ runs. These metrics provide useful insights into memory usage and other runtime metrics of the JVM.

The JMX Plugin enables JMX monitoring for HiveMQ. When the plugin is loaded, you can use JMX monitoring tools such as JConsole to get statistics and insights for your HiveMQ deployment.

Configuration

If you run HiveMQ behind NAT (Network Address Translation), you need to set some additional options:

JAVA_OPTS="$JAVA_OPTS -Djava.rmi.server.hostname=<PUBLIC_IP>"
JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.rmi.port=9010"

This configuration allows you to connect with JConsole using PUBLIC_IP:9010.

MBeans

When JMX is activated, the following MBeans (managed Java objects) are available for monitoring:

MBean Name Description

metrics

The HiveMQ metrics and statistics. A list of all available metrics is available here.

java.nio

Statistics and metrics about used native memory.

java.lang

All information about the Java Virtual Machine.

actions

Actions that can be started via JMX.

Graphite

Graphite is a graphing system for monitoring and displaying statistics from different data sources. Graphite is highly scalable and useful when you need to monitor one or more HiveMQ cluster nodes. For example, if the built-in JMX Monitoring is not sufficient for your use case or you want to preserve statistics history.

Graphite Server
We strongly recommend that you install Graphite on a different server than HiveMQ.

Graphite monitoring is not part of the HiveMQ installation. You can download a free plugin that enables HiveMQ to report to Graphite on GitHub.

Maintenance

Maintenance actions are often resource intensive and should therefore not be started when HiveMQ instances have been added to the cluster or removed recently. Before starting a maintenance action, make sure the actual cluster size is equal to the supposed cluster size.

Maintenance actions Description

subscription cleanup

Clean up outdated subscriptions that were added during network splits.

$SYS Topic

The free and open source HiveMQ SYS Topic Plugin enables HiveMQ to support a special MQTT topic tree called $SYS Topic. MQTT clients can subscribe to this topic and receive system information and statistics of the HiveMQ MQTT broker. Once the plugin is installed, the following $SYS subtopics are exposed:

Table 1. $SYS subtopics
Topic Description

$SYS/broker/clients/connected

The currently connected clients.

$SYS/broker/clients/disconnected

The clients which are not connected and have a persistent session on the broker.

$SYS/clients/maximum

The maximum number of active clients which were connected simultaneously.

$SYS/clients/total

The total count of connected and disconnect (with persistent session) clients.

$SYS/load/bytes/received

The total bytes received.

$SYS/load/bytes/sent

The total bytes sent.

$SYS/broker/load/connections/1min

The moving average of the number of CONNECT packets received by the broker during the last minute.

$SYS/broker/load/connections/5min

The moving average of the number of CONNECT packets received by the broker during the last 5 minutes.

$SYS/broker/load/connections/15min

The moving average of the number of CONNECT packets received by the broker during the last 15 minutes.

$SYS/broker/load/messages/received/1min

The moving average of the number of all types of MQTT messages received by the broker during the last minute.

$SYS/broker/load/messages/received/5min

The moving average of the number of all types of MQTT messages received by the broker during the last 5 minutes.

$SYS/broker/load/messages/received/15min

The moving average of the number of all types of MQTT messages received by the broker during the last 15 minutes.

$SYS/broker/load/messages/sent/1min

The moving average of the number of all types of MQTT messages sent by the broker during the last minute.

$SYS/broker/load/messages/sent/5min

The moving average of the number of all types of MQTT messages sent by the broker during the last 5 minutes.

$SYS/broker/load/messages/sent/15min

The moving average of the number of all types of MQTT messages sent by the broker during the last 15 minutes.

$SYS/broker/load/publish/dropped/1min

The moving average of the number of MQTT PUBLISH messages dropped by the broker during the last minute.

$SYS/broker/load/publish/dropped/5min

The moving average of the number of MQTT PUBLISH messages dropped by the broker during the last 5 minutes.

$SYS/broker/load/publish/dropped/15min

The moving average of the number of MQTT PUBLISH messages dropped by the broker during the last 15 minutes.

$SYS/broker/load/publish/received/1min

The moving average of the number of MQTT PUBLISH messages received by the broker during the last minute.

$SYS/broker/load/publish/received/5min

The moving average of the number of MQTT PUBLISH messages received by the broker during the last 5 minutes.

$SYS/broker/load/publish/received/15min

The moving average of the number of MQTT PUBLISH messages received by the broker during the last 15 minutes.

$SYS/broker/load/publish/sent/1min

The moving average of the number of MQTT PUBLISH messages sent by the broker during the last minute.

$SYS/broker/load/publish/sent/5min

The moving average of the number of MQTT PUBLISH messages sent by the broker during the last 5 minutes.

$SYS/broker/load/publish/sent/15min

The moving average of the number of MQTT PUBLISH messages sent by the broker during the last 15 minutes.

$SYS/broker/messages/publish/dropped

The total number of MQTT PUBLISH messages that have been dropped due to inflight/queuing limits.

$SYS/broker/messages/publish/received

The total MQTT PUBLISH messages received.

$SYS/broker/messages/publish/sent

The total MQTT PUBLISH messages sent.

$SYS/broker/messages/received

The total MQTT messages received.

$SYS/broker/messages/retained/count

The amount of all retained messages.

$SYS/broker/messages/sent

The total MQTT messages sent.

$SYS/broker/subscriptions/count

The total count of subscriptions

$SYS/broker/time

The current time on the broker. Only published on subscription.

$SYS/broker/uptime

The uptime of the broker in seconds. Only published on subscription.

$SYS/broker/version

The HiveMQ version. Only published on subscription.

$SYS Topic standard
There is no official standardization what $SYS topics should exist. There is a consensus between broker vendors of available $SYS topics available here. These special topics are not always fully interoperable between MQTT brokers and clients should not rely on these topics.

It is not possible for any client to publish to the $SYS topic or one of its subtopics. These values are published exclusively by HiveMQ.

While $SYS topics are a good fit for broker monitoring in a trusted environment, we recommend not using SYS topics in production and relying on a more sophisticated monitoring solution.

Available Metrics

Metric Types

There are five different types of Metrics available. The following table shows all available metric types:

Table 2. Metric Types
Metric Type Description

Gauge

A gauge returns a simple value at the point of time the metric was requested

Counter

A counter is a simple incrementing and decrementing number.

Histogram

A histogram measures the distribution of values in a stream of data. They allow to measure min, mean, max, standard deviation of values and quantiles.

Meter

A meter measures the rate at which a set of events occur. Meters measure mean, 1-, 5-, and 15-minute moving averages of events.

Timer

A timer is basically a histogram of the duration of a type of event and a meter of the rate of its occurrence. It captures rate and duration information.

The following table lists metrics that are available for monitoring HiveMQ regardless if the HiveMQ server instance runs in single mode or as part of a cluster:

Table 3. Available standard metric details
Metric Type Description

com.hivemq.cache.payload-persistence.averageLoadPenalty

Gauge

Cache statistic capturing the average load penalty of the payload persistence cache

com.hivemq.cache.payload-persistence.evictionCount

Gauge

Cache statistic capturing the eviction count of the payload persistence cache

com.hivemq.cache.payload-persistence.hitCount

Gauge

Cache statistic capturing the hit count of the payload persistence cache

com.hivemq.cache.payload-persistence.hitRate

Gauge

Cache statistic capturing the hit rate of the payload persistence cache

com.hivemq.cache.payload-persistence.loadCount

Gauge

Cache statistic capturing the load count of the payload persistence cache

com.hivemq.cache.payload-persistence.loadExceptionCount

Gauge

Cache statistic capturing the load exception count of the payload persistence cache

com.hivemq.cache.payload-persistence.loadExceptionRate

Gauge

Cache statistic capturing the load exception rate of the payload persistence cache

com.hivemq.cache.payload-persistence.loadSuccessCount

Gauge

Cache statistic capturing the load success count of the payload persistence cache

com.hivemq.cache.payload-persistence.missCount

Gauge

Cache statistic capturing the miss count of the payload persistence cache

com.hivemq.cache.payload-persistence.missRate

Gauge

Cache statistic capturing the miss rate of the payload persistence cache

com.hivemq.cache.payload-persistence.requestCount

Gauge

Cache statistic capturing the request count of the payload persistence cache

com.hivemq.cache.payload-persistence.totalLoadTime

Gauge

Cache statistic capturing the total load time of the payload persistence cache

com.hivemq.cache.shared-subscription.averageLoadPenalty

Gauge

Cache statistic capturing the average load penalty of the shared subscription cache

com.hivemq.cache.shared-subscription.evictionCount

Gauge

Cache statistic capturing the eviction count of the shared subscription cache

com.hivemq.cache.shared-subscription.hitCount

Gauge

Cache statistic capturing the hit count of the shared subscription cache

com.hivemq.cache.shared-subscription.hitRate

Gauge

Cache statistic capturing the hit rate of the shared subscription cache

com.hivemq.cache.shared-subscription.loadCount

Gauge

Cache statistic capturing the load count of the shared subscription cache

com.hivemq.cache.shared-subscription.loadExceptionCount

Gauge

Cache statistic capturing the load exception count of the shared subscription cache

com.hivemq.cache.shared-subscription.loadExceptionRate

Gauge

Cache statistic capturing the load exception rate of the shared subscription cache

com.hivemq.cache.shared-subscription.loadSuccessCount

Gauge

Cache statistic capturing the load success count of the shared subscription cache

com.hivemq.cache.shared-subscription.missCount

Gauge

Cache statistic capturing the miss count of the shared subscription cache

com.hivemq.cache.shared-subscription.missRate

Gauge

Cache statistic capturing the miss rate of the shared subscription cache

com.hivemq.cache.shared-subscription.requestCount

Gauge

Cache statistic capturing the request count of the shared subscription cache

com.hivemq.cache.shared-subscription.totalLoadTime

Gauge

Cache statistic capturing the total load time of the shared subscription cache

com.hivemq.callback.executor.completed

Meter

Measures the current rate of completed CallbackExecutor jobs

com.hivemq.callback.executor.duration

Timer

Captures metrics about the job durations for jobs submitted to the CallbackExecutor

com.hivemq.callback.executor.running

Counter

Measures how many CallbackExecutor jobs are running at the moment

com.hivemq.callback.executor.submitted

Meter

Measures the current rate of submitted jobs to the CallbackExecutor

com.hivemq.clients.half-full-queue.count

Counter

Counts the amount of clients with an inflight queue, which is at least half-full

com.hivemq.cluster.name-request.retry.count

Counter

The number of retry attempts that are processed to resolve the name of a node

com.hivemq.cluster.nodes.count

Gauge

Captures the current number of nodes in the cluster

com.hivemq.cluster.topology-change.time

Timer

Measures the time spent waiting for cluster topology changes

com.hivemq.direct-memory.used

Gauge

Currently used direct memory in bytes

com.hivemq.exceptions.total

Meter

Measures the rate of inconsequential exceptions thrown during the socket life cycle

com.hivemq.keep-alive.disconnect.count

Counter

Counts every closed connection that was closed because the client missed sending PINGREQ message during the keep-alive interval

com.hivemq.logging.all

Meter

Measures the rate of logging statements of all levels

com.hivemq.logging.debug

Meter

Measures the rate of logging statements in DEBUG level

com.hivemq.logging.error

Meter

Measures the rate of logging statements in ERROR level

com.hivemq.logging.info

Meter

Measures the rate of logging statements in INFO level

com.hivemq.logging.trace

Meter

Measures the rate of logging statements in TRACE level

com.hivemq.logging.warn

Meter

Measures the rate of logging statements in WARN level

com.hivemq.messages.dropped.count

Counter

Counts every dropped message.

com.hivemq.messages.dropped.in-flight-window.count

Counter

Counts the messages that have been dropped because the in flight window was full

com.hivemq.messages.dropped.internal-error.count

Counter

Counts the messages that have been dropped due to internal errors

com.hivemq.messages.dropped.not-connected.count

Counter

Counts the messages that have been dropped because the client disconnected and has no persistent session

com.hivemq.messages.dropped.not-writable.count

Counter

Counts the messages with QoS 0 that have been dropped because the client socket was not writeable

com.hivemq.messages.dropped.qos-0-queue-not-empty.count

Counter

Counts the messages with QoS 0 that have been dropped because the queue for the client wasn’t empty

com.hivemq.messages.dropped.queue-full.count

Counter

Counts the messages that have been dropped because the client session message queue was full

com.hivemq.messages.dropped.rate

Meter

Measures the current rate of dropped messages.

com.hivemq.messages.incoming.connect.count

Counter

Counts every incoming MQTT CONNECT message

com.hivemq.messages.incoming.connect.rate

Meter

Measures the current rate of incoming MQTT CONNECT messages

com.hivemq.messages.incoming.disconnect.count

Counter

Counts every incoming MQTT DISCONNECT message

com.hivemq.messages.incoming.disconnect.rate

Meter

Measures the current rate of incoming MQTT DISCONNECT messages

com.hivemq.messages.incoming.pingreq.count

Counter

Counts every incoming MQTT PINGREQ message

com.hivemq.messages.incoming.pingreq.rate

Meter

Measures the current rate of incoming MQTT PINGREQ messages

com.hivemq.messages.incoming.puback.count

Counter

Counts every incoming MQTT PUBACK message

com.hivemq.messages.incoming.puback.rate

Meter

Measures the current rate of incoming MQTT PUBACK messages

com.hivemq.messages.incoming.pubcomp.count

Counter

Counts every incoming MQTT PUBCOMP message

com.hivemq.messages.incoming.pubcomp.rate

Meter

Measures the current rate of incoming MQTT PUBCOMP messages

com.hivemq.messages.incoming.publish.bytes

Histogram

Measures the distribution of incoming MQTT message size (including MQTT packet headers)

com.hivemq.messages.incoming.publish.count

Counter

Counts every incoming MQTT PUBLISH message

com.hivemq.messages.incoming.publish.rate

Meter

Measures the current rate of incoming MQTT PUBLISH messages

com.hivemq.messages.incoming.pubrec.count

Counter

Counts every incoming MQTT PUBREC message

com.hivemq.messages.incoming.pubrec.rate

Meter

Measures the current rate of incoming MQTT PUBREC messages

com.hivemq.messages.incoming.pubrel.count

Counter

Counts every incoming MQTT PUBREL message

com.hivemq.messages.incoming.pubrel.rate

Meter

Measures the current rate of incoming MQTT PUBREL messages

com.hivemq.messages.incoming.subscribe.count

Counter

Counts every incoming MQTT SUBSCRIBE message

com.hivemq.messages.incoming.subscribe.rate

Meter

Measures the current rate of incoming MQTT SUBSCRIBE messages

com.hivemq.messages.incoming.total.bytes

Histogram

Measures the size distribution of incoming MQTT messages (including MQTT packet headers)

com.hivemq.messages.incoming.total.count

Counter

Counts every incoming MQTT message

com.hivemq.messages.incoming.total.rate

Meter

Measures the current rate of incoming MQTT messages

com.hivemq.messages.incoming.unsubscribe.count

Counter

Counts every incoming MQTT UNSUBSCRIBE message

com.hivemq.messages.incoming.unsubscribe.rate

Meter

Measures the current rate of incoming MQTT UNSUBSCRIBE messages

com.hivemq.messages.outgoing.connack.count

Counter

Counts every outgoing MQTT CONNACK message

com.hivemq.messages.outgoing.connack.rate

Meter

Measures the current rate of outgoing MQTT CONNACK messages

com.hivemq.messages.outgoing.pingresp.count

Counter

Counts every outgoing MQTT PINGRESP message

com.hivemq.messages.outgoing.pingresp.rate

Meter

Measures the current rate of outgoing MQTT PINGRESP messages

com.hivemq.messages.outgoing.puback.count

Counter

Counts every outgoing MQTT PUBACK message

com.hivemq.messages.outgoing.puback.rate

Meter

Measures the current rate of outgoing MQTT PUBACK messages

com.hivemq.messages.outgoing.pubcomp.count

Counter

Counts every outgoing MQTT PUBCOMP message

com.hivemq.messages.outgoing.pubcomp.rate

Meter

Measures the current rate of outgoing MQTT PUBCOMP messages

com.hivemq.messages.outgoing.publish.bytes

Histogram

Measures the size distribution of outgoing MQTT messages (including MQTT packet headers)

com.hivemq.messages.outgoing.publish.count

Counter

Counts every outgoing MQTT PUBLISH message

com.hivemq.messages.outgoing.publish.rate

Meter

Measures the current rate of outgoing MQTT PUBLISH messages

com.hivemq.messages.outgoing.pubrec.count

Counter

Counts every outgoing MQTT PUBREC message

com.hivemq.messages.outgoing.pubrec.rate

Meter

Measures the current rate of outgoing MQTT PUBREC messages

com.hivemq.messages.outgoing.pubrel.count

Counter

Counts every outgoing MQTT PUBREL message

com.hivemq.messages.outgoing.pubrel.rate

Meter

Measures the current rate of outgoing MQTT PUBREL messages

com.hivemq.messages.outgoing.suback.count

Counter

Counts every outgoing MQTT SUBACK message

com.hivemq.messages.outgoing.suback.rate

Meter

Measures the current rate of outgoing MQTT SUBACK messages

com.hivemq.messages.outgoing.total.bytes

Histogram

Measures the size distribution of outgoing MQTT messages (including MQTT packet headers)

com.hivemq.messages.outgoing.total.count

Counter

Counts every outgoing MQTT message

com.hivemq.messages.outgoing.total.rate

Meter

Measures the current rate of outgoing MQTT messages

com.hivemq.messages.outgoing.unsuback.count

Counter

Counts every outgoing MQTT UNSUBACK message

com.hivemq.messages.outgoing.unsuback.rate

Meter

Measures the current rate of outgoing MQTT UNSUBACK messages

com.hivemq.messages.publish-resent

Meter

Measures the current rate of resent PUBLISH messages (QoS > 0)

com.hivemq.messages.pubrel-resent

Meter

Measures the current rate of resent PUBREL messages (OoS = 2)

com.hivemq.messages.retained.current

Gauge

The current amount of retained messages

com.hivemq.messages.retained.mean

Histogram

Metrics about the mean payload-size of retained messages in bytes

com.hivemq.messages.retained.rate

Meter

The current rate of newly retained messages

com.hivemq.networking.bytes.read.current

Gauge

The current (last 5 seconds) amount of read bytes

com.hivemq.networking.bytes.read.total

Gauge

The total amount of read bytes

com.hivemq.networking.bytes.write.current

Gauge

The current (last 5 seconds) amount of written bytes

com.hivemq.networking.bytes.write.total

Gauge

Total amount of written bytes

com.hivemq.networking.connections.current

Gauge

The current total number of active MQTT connections

com.hivemq.networking.connections.mean

Histogram

The mean total number of active MQTT connections

com.hivemq.networking.connections-closed.graceful.count

Counter

Counts clients which disconnected after sending a DISCONNECT Message

com.hivemq.networking.connections-closed.ungraceful.count

Counter

Counts clients which disconnected without sending a DISCONNECT Message

com.hivemq.networking.connections-closed.total.count

Counter

Counts all clients which disconnected from HiveMQ (= graceful + ungraceful)

com.hivemq.payload-persistence.cleanup-executor.completed

Meter

Measure the rate of completed tasks submitted to the scheduler in charge of the cleanup of the persistence payload

com.hivemq.payload-persistence.cleanup-executor.duration

Timer

Captures metrics about the job durations for jobs submitted to the scheduler in charge of the cleanup of the persistence payload

com.hivemq.payload-persistence.cleanup-executor.running

Counter

Counts tasks that are currently running in the scheduler in charge of the cleanup of the persistence payload

com.hivemq.payload-persistence.cleanup-executor.scheduled.once

Meter

Measures about the tasks that have been scheduled to run only once in the scheduler in charge of the cleanup of the persistence payload

com.hivemq.payload-persistence.cleanup-executor.scheduled.overrun

Counter

Counts the periodic tasks which ran longer then their time frame allowed in the scheduler in charge of the cleanup of the persistence payload

com.hivemq.payload-persistence.cleanup-executor.scheduled.percent-of-period

Histogram

Metrics about how much percent of their allowed time frame periodic tasks used while running the cleanup of the persistence payload

com.hivemq.payload-persistence.cleanup-executor.scheduled.repetitively

Meter

Measures about the tasks that have been scheduled to run repetitively in the scheduler in charge of the cleanup of the persistence payload

com.hivemq.payload-persistence.cleanup-executor.submitted

Meter

Measures about the tasks that have been submitted to the scheduler in charge of the cleanup of the persistence payload

com.hivemq.persistence-executor.completed

Meter

Measure the rate of completed tasks submitted to the persistence executor

com.hivemq.persistence-executor.duration

Timer

Captures metrics about the job durations for jobs submitted to the persistence executor

com.hivemq.persistence-executor.running

Counter

Counts tasks that are currently running in the persistence executor

com.hivemq.persistence-executor.submitted

Meter

Measures about the tasks that have been submitted to the scheduler responsible for persistence

com.hivemq.persistence-scheduled-executor.completed

Meter

Measure the rate of completed tasks submitted to the scheduler responsible for persistence

com.hivemq.persistence-scheduled-executor.duration

Timer

Captures metrics about the job durations for jobs submitted to the scheduler responsible for persistence

com.hivemq.persistence-scheduled-executor.running

Counter

Counts tasks that are currently running in the scheduler responsible for persistence

com.hivemq.persistence-scheduled-executor.scheduled.once

Meter

Measures about the tasks that have been scheduled to run once in the scheduler responsible for persistence

com.hivemq.persistence-scheduled-executor.scheduled.overrun

Counter

Counts the periodic tasks which ran longer then their time frame allowed in the scheduler responsible for persistence

com.hivemq.persistence-scheduled-executor.scheduled.percent-of-period

Histogram

Metrics about how much percent of their allowed time frame periodic tasks used in the scheduler responsible for persistence

com.hivemq.persistence-scheduled-executor.scheduled.repetitively

Meter

Measures about the tasks that have been scheduled to run repetitively in the scheduler responsible for persistence

com.hivemq.persistence-scheduled-executor.submitted

Meter

Measures about the tasks that have been submitted to the scheduler responsible for persistence

com.hivemq.persistence.executor.client-session.tasks

Gauge

Current amount of disk I/O tasks that are enqueued by the client session persistence

com.hivemq.persistence.executor.client-session.time

Timer

Measures the mean execution time (in nanoseconds) of client session disk I/O tasks

com.hivemq.persistence.executor.noempty-queues

Gauge

Current amount of single writer task queues that are not empty

com.hivemq.persistence.executor.outgoing-message-flow.tasks

Gauge

Current amount of disk I/O tasks that are enqueued by the outgoing message flow persistence

com.hivemq.persistence.executor.outgoing-message-flow.time

Timer

Measures the mean execution time (in nanoseconds) of outgoing message flow disk I/O tasks

com.hivemq.persistence.executor.queue-misses

Counter

Current count of loops that all single writer threads have done without executing a task

com.hivemq.persistence.executor.queued-messages.tasks

Gauge

Current amount of disk I/O tasks that are enqueued by the queued messages persistence

com.hivemq.persistence.executor.queued-messages.time

Timer

Measures the mean execution time (in nanoseconds) of queued messages disk I/O tasks

com.hivemq.persistence.executor.request-event-bus.tasks

Gauge

Current amount of tasks that are enqueued by the request event bus

com.hivemq.persistence.executor.request-event-bus.time

Timer

Measures the mean execution time (in nanoseconds) of request event bus tasks

com.hivemq.persistence.executor.retained-messages.tasks

Gauge

Current amount of disk I/O tasks that are enqueued by the retained message persistence

com.hivemq.persistence.executor.retained-messages.time

Timer

Measures the mean execution time (in nanoseconds) of retained message disk I/O tasks

com.hivemq.persistence.executor.running.threads

Gauge

Current amount of threads that are executing disk I/O tasks

com.hivemq.persistence.executor.subscription.tasks

Gauge

Current amount of disk I/O tasks that are enqueued by the subscription persistence

com.hivemq.persistence.executor.subscription.time

Timer

Measures the mean execution time (in nanoseconds) of subscription disk I/O tasks

com.hivemq.persistence.executor.total.tasks

Gauge

Current amount of disk I/O tasks that are enqueued by all persistence executors

com.hivemq.persistence.payload-entries.count

Gauge

Holds the current amount of payloads stored in the payload persistence

com.hivemq.persistence.removable-entries.count

Gauge

Holds the current amount of payloads stored in the payload persistence, that can be removed by the cleanup

com.hivemq.plugin.callbacks.after-login.failed.time

Timer

Metrics about the AfterLoginCallback

com.hivemq.plugin.callbacks.after-login.success.time

Timer

Metrics about the AfterLoginCallback

com.hivemq.plugin.callbacks.authentication.time

Timer

Metrics about the OnAuthenticationCallback

com.hivemq.plugin.callbacks.authorization.time

Timer

Metrics about the OnAuthorizationCallback

com.hivemq.plugin.callbacks.connack-send.time

Timer

Metrics about the OnConnackSend Callback

com.hivemq.plugin.callbacks.connect.time

Timer

Metrics about the OnConnectCallback

com.hivemq.plugin.callbacks.disconnect.time

Timer

Metrics about the OnDisconnectCallback

com.hivemq.plugin.callbacks.permissions-disconnect.publish.time

Timer

Metrics about the OnInsufficientPermissionDisconnectCallback

com.hivemq.plugin.callbacks.permissions-disconnect.subscribe.time

Timer

Metrics about the OnInsufficientPermissionDisconnectCallback

com.hivemq.plugin.callbacks.ping.time

Timer

Metrics about the OnPingCallback

com.hivemq.plugin.callbacks.puback-received.time

Timer

Metrics about the OnPubackReceived Callback

com.hivemq.plugin.callbacks.puback-send.time

Timer

Metrics about the OnPubackSend Callback

com.hivemq.plugin.callbacks.pubcomp-received.time

Timer

Metrics about the OnPubcompReceived Callback

com.hivemq.plugin.callbacks.pubcomp-send.time

Timer

Metrics about the OnPubcompSend Callback

com.hivemq.plugin.callbacks.publish-received.time

Timer

Metrics about the OnPublishReceivedCallback

com.hivemq.plugin.callbacks.publish-send.time

Timer

Metrics about the OnPublishSend Callback

com.hivemq.plugin.callbacks.pubrec-received.time

Timer

Metrics about the OnPubrecSend Callback

com.hivemq.plugin.callbacks.pubrec-send.time

Timer

Metrics about the OnPubrecSend Callback

com.hivemq.plugin.callbacks.pubrel-received.time

Timer

Metrics about the OnPubrelReceived Callback

com.hivemq.plugin.callbacks.pubrel-send.time

Timer

Metrics about the OnPubrelSend Callback

com.hivemq.plugin.callbacks.restrictions.time

Timer

Metrics about the RestrictionsAfterLoginCallback

com.hivemq.plugin.callbacks.suback-send.time

Timer

Metrics about the OnSubackSend Callback

com.hivemq.plugin.callbacks.subscribe.time

Timer

Metrics about the OnSubscribeCallback

com.hivemq.plugin.callbacks.topic-subscription.time

Timer

Metrics about the OnTopicSubscriptionCallback

com.hivemq.plugin.callbacks.unsuback-send.time

Timer

Metrics about the OnUnsubackSend Callback

com.hivemq.plugin.callbacks.unsubscribe.time

Timer

Metrics about the OnUnsubscribeReceivedCallback

com.hivemq.plugin.executor.completed

Meter

Measure the rate of completed tasks submitted to the plugin executor

com.hivemq.plugin.executor.duration

Timer

Measure the rate of completed tasks submitted to the scheduler responsible for plugins

com.hivemq.plugin.executor.running

Counter

Counts tasks that are currently running in the scheduler responsible for plugins

com.hivemq.plugin.executor.scheduled.once

Meter

Measures about the tasks that have been scheduled to run once in the scheduler responsible for plugins

com.hivemq.plugin.executor.scheduled.overrun

Counter

Counts the periodic tasks which ran longer then their time frame allowed in the scheduler responsible for plugins

com.hivemq.plugin.executor.scheduled.percent-of-period

Histogram

Metrics about how much percent of their allowed time frame periodic tasks used in the scheduler responsible for plugins

com.hivemq.plugin.executor.scheduled.repetitively

Meter

Measures about the tasks that have been scheduled to run repetitively in the scheduler responsible for plugins

com.hivemq.plugin.executor.submitted

Meter

Measures about the tasks that have been submitted to the scheduler responsible for plugins

com.hivemq.queues.publish.rate

Meter

Measures the rate of messages put into the publish queue

com.hivemq.queues.publish.size

Counter

Measures the current count of messages in the publish queue

com.hivemq.sessions.overall.current

Gauge

Measures the current count of stored sessions. These sessions include all sessions, including online and offline clients

com.hivemq.sessions.persistent.active

Counter

Measures the current count of active persistent sessions (= Online MQTT clients which are connected with cleanSession=false).

com.hivemq.single-writer-executor.completed

Meter

Measure the rate of completed tasks submitted to the single-writer executor

com.hivemq.single-writer-executor.duration

Timer

Measure the rate of completed tasks submitted to the scheduler responsible for single-writer

com.hivemq.single-writer-executor.running

Counter

Counts tasks that are currently running in the scheduler responsible for single-writer

com.hivemq.single-writer-executor.submitted

Meter

Measures about the tasks that have been submitted to the scheduler responsible for single-writer

com.hivemq.subscriptions.overall.current

Counter

Measures the current count of subscriptions on the broker

Table 4. System and operating system metrics
Metric Type Description

com.hivemq.system.max-file-descriptor

Gauge

Maximum allowed amount of file descriptors as seen by the JVM

com.hivemq.system.open-file-descriptor

Gauge

Amount of open file descriptors as seen by the JVM

com.hivemq.system.physical-memory.free

Gauge

Current amount of free physical memory in bytes

com.hivemq.system.physical-memory.total

Gauge

Total amount of physical memory (bytes) available

com.hivemq.system.process-cpu.load

Gauge

Current CPU usage for the JVM process (0.0 idle – 1.0 full CPU usage)

com.hivemq.system.process-cpu.time

Gauge

Total amount of CPU time the JVM process has used to this point(in nanoseconds)

com.hivemq.system.swap-space.free

Gauge

Current amount of free swap space in bytes

com.hivemq.system.swap-space.total

Gauge

Total amount of swap space available in bytes

com.hivemq.system.system-cpu.load

Gauge

Current CPU usage for the whole system (0.0 idle – 1.0 full CPU usage)

com.hivemq.system.os.global.uptime

Gauge

OS Uptime in seconds

com.hivemq.system.os.global.memory.total

Gauge

The amount of actual physical memory, in bytes

com.hivemq.system.os.global.memory.available

Gauge

The amount of physical memory currently available, in bytes.

com.hivemq.system.os.global.memory.swap.used

Gauge

The current memory committed to the paging/swap file(s), in bytes

com.hivemq.system.os.global.memory.swap.total

Gauge

The current size of the paging/swap file(s), in bytes.

com.hivemq.system.os.file-descriptors.open

Gauge

Amount of currently open file descriptors

com.hivemq.system.os.file-descriptors.max

Gauge

Maximum allowed amount of file descriptors

com.hivemq.system.os.global.cpu.<cpu-name>.usage.idle

Gauge

Percentage of time that the CPU or CPUs were idle and the system did not have an outstanding disk I/O request

com.hivemq.system.os.global.cpu.<cpu-name>.usage.iowait

Gauge

Percentage of time that the CPU or CPUs were idle during which the system had an outstanding disk I/O request.

com.hivemq.system.os.global.cpu.<cpu-name>.usage.irq

Gauge

Percentage of time that the CPU used to service hardware IRQs

com.hivemq.system.os.global.cpu.<cpu-name>.usage.nice

Gauge

Percentage of CPU utilization that occurred while executing at the user level with nice priority

com.hivemq.system.os.global.cpu.<cpu-name>.usage.softirq

Gauge

Percentage of time that the CPU used to service soft IRQs

com.hivemq.system.os.global.cpu.<cpu-name>.usage.steal

Gauge

Percentage of time which the hypervisor dedicated for other guests in the system.

com.hivemq.system.os.global.cpu.<cpu-name>.usage.sys

Gauge

Percentage of CPU utilization that occurred while executing at the system level (kernel)

com.hivemq.system.os.global.cpu.<cpu-name>.usage.user

Gauge

Percentage of CPU utilization that occurred while executing at the user level (application)

com.hivemq.system.os.global.cpu.<cpu-name>.total

Gauge

Percentage of total CPU utilization for convenience (not idle, calculated as sum of usage values)

com.hivemq.system.os.process.threads.count

Gauge

Number of threads of the HiveMQ process as seen by the OS

com.hivemq.system.os.process.time-spent.user

Gauge

Amount of milliseconds the the HiveMQ process has executed in user mode as seen by the OS

com.hivemq.system.os.process.time-spent.kernel

Gauge

Amount of milliseconds the HiveMQ process has executed in kernel/system mode as seen by the OS

com.hivemq.system.os.process.memory.virtual

Gauge

Virtual Memory Size (VSZ) in bytes. It includes all memory that the HiveMQ process can access, including memory that is swapped out and memory that is from shared libraries.

com.hivemq.system.os.process.memory.resident-set-size

Gauge

Resident Set Size (RSS) in bytes. It is used to show how much memory is allocated to the HiveMQ process and is in RAM. It does not include memory that is swapped out. It does include memory from shared libraries as long as the pages from those libraries are actually in memory. It does include all stack and heap memory.

com.hivemq.system.os.process.disk.bytes-written

Gauge

Number of bytes the HiveMQ process has written to disk

com.hivemq.system.os.process.disk.bytes-read

Gauge

Number of bytes the HiveMQ process has read from disk

com.hivemq.system.os.network-interface.<if-name>.bytes-received

Gauge

Bytes received by the network interface <if-name>

com.hivemq.system.os.network-interface.<if-name>.bytes-sent

Gauge

Bytes sent by the network interface <if-name>

com.hivemq.system.os.network-interface.<if-name>.packets-sent

Gauge

Packets sent by the network interface <if-name>

com.hivemq.system.os.network-interface.<if-name>.packets-received

Gauge

Packets received by the network interface <if-name>

com.hivemq.system.os.network-interface.<if-name>.in-errors

Gauge

Input errors for the network interface <if-name>

com.hivemq.system.os.network-interface.<if-name>.out-errors

Gauge

Output errors for the network interface <if-name>

com.hivemq.system.os.disks.<disk-name>.size

Gauge

Total size of disk with name <disk-name> in bytes

com.hivemq.system.os.disks.<disk-name>.read.count

Gauge

Amount of read operations for disk with name <disk-name>

com.hivemq.system.os.disks.<disk-name>.write.count

Gauge

Amount of read operations for disk with name <disk-name>

com.hivemq.system.os.disks.<disk-name>.read.bytes

Gauge

Amount of bytes read from disk with name <disk-name>

com.hivemq.system.os.disks.<disk-name>.write.bytes

Gauge

Amount of bytes written from disk with name <disk-name>

The following table (Table 4) lists metrics that are only available for monitoring if the HiveMQ server instance is part of a cluster.

Table 5. Additional metric in cluster environment
Metric Type Description

com.hivemq.cluster.sent.*

Meter

Provides measures for every class that made once a SEND request (every class get its own metric)

Monitoring dropped messages

In a healthy MQTT environment, messages should never be dropped, especially if your MQTT clients rely on receiving important messages.

It’s highly recommended to monitor if messages are dropped or if the average queued messages are at a critical number.

The following metrics of HiveMQ can be used to monitor the drop rate (and total count) of messages and the average queued message usage in the system.

Table 6. Monitoring Metrics for Dropped Messages
Metric Description

com.hivemq.messages.dropped.rate

The rate of dropped messages per second. This metric also exposes a total counter of dropped messages.

com.hivemq.clients.half-full-queue.count

The number of clients with an inflight queue, which is at least half-full.

We recommend that the average number of queued messages should not exceed 50% of the maximum amount queued messages for a longer time if you want to prevent message drop.

Monitoring of Plugins

With the powerful HiveMQ Plugin System it is possible to add integration plugins for virtually everything you can imagine. A common pitfall when writing plugins is that these plugins block HiveMQ threads in some way and the overall performance of the installation can decrease dramatically. Fortunately HiveMQ offers a way to monitor the execution time of plugin callback on specific callbacks.

The following metrics can be monitored (e.g. with JMX) for plugin callback execution times:

Table 7. Available plugin callback execution time metrics
Metric Name Description

com.hivemq.plugin.callbacks.after-login.failed.time

Metrics about the AfterLoginCallback

com.hivemq.plugin.callbacks.after-login.success.time

Metrics about the AfterLoginCallback

com.hivemq.plugin.callbacks.authentication.time

Metrics about the OnAuthenticationCallback

com.hivemq.plugin.callbacks.authorization.time

Metrics about the OnAuthorizationCallback

com.hivemq.plugin.callbacks.connack-send.time

Metrics about the OnConnackSend Callback

com.hivemq.plugin.callbacks.connect.time

Metrics about the OnConnectCallback

com.hivemq.plugin.callbacks.disconnect.time

Metrics about the OnDisconnectCallback

com.hivemq.plugin.callbacks.permissions-disconnect.publish.time

Metrics about the OnInsufficientPermissionDisconnect Callback

com.hivemq.plugin.callbacks.permissions-disconnect.subscribe.time

Metrics about the OnInsufficientPermissionDisconnect Callback

com.hivemq.plugin.callbacks.puback-received.time

Metrics about the OnPubackReceived Callback

com.hivemq.plugin.callbacks.puback-send.time

Metrics about the OnPubackSend Callback

com.hivemq.plugin.callbacks.pubcomp-received.time

Metrics about the OnPubcompReceived Callback

com.hivemq.plugin.callbacks.pubcomp-send.time

Metrics about the OnPubcompSend Callback

com.hivemq.plugin.callbacks.publish-received.time

Metrics about the OnPublishReceivedCallback

com.hivemq.plugin.callbacks.publish-send.time

Metrics about the OnPublishSend Callback

com.hivemq.plugin.callbacks.pubrec-received.time

Metrics about the OnPubrecReceived Callback

com.hivemq.plugin.callbacks.pubrec-send.time

Metrics about the OnPubrecSend Callback

com.hivemq.plugin.callbacks.pubrel-received.time

Metrics about the OnPubreReceived Callback

com.hivemq.plugin.callbacks.pubrel-send.time

Metrics about the OnPubrelSend Callback

com.hivemq.plugin.callbacks.restrictions.time

Metrics about the RestrictionsAfterLoginCallback

com.hivemq.plugin.callbacks.suback-send.time

Metrics about the OnSubackSend Callback

com.hivemq.plugin.callbacks.subscribe.time

Metrics about the OnSubscribeCallback

com.hivemq.plugin.callbacks.unsuback-send.time

Metrics about the OnUnsubackSend Callback

com.hivemq.plugin.callbacks.unsubscribe.time

Metrics about the OnUnsubscribeReceivedCallback

For all these metrics you can get the following details

Table 8. Available metric details
Attribute Name Description

50thPercentile

The 50th percentile for callback execution times

75thPercentile

The 75th percentile for callback execution times

95thPercentile

The 95th percentile for callback execution times

98thPercentile

The 98th percentile for callback execution times

99thPercentile

The 99th percentile for callback execution times

999thPercentile

The 99.9th percentile for callback execution times

Mean

The mean for callback execution times

StdDev

The standard deviation of callback execution times

Count

The total count of callback executions

FifteenMinuteRate

The average rate (events/s) of callback executions in the last 15 minutes

FiveMinuteRate

The average rate (events/s) of callback executions in the last 5 minutes

OneMinuteRate

The average rate (events/s) of callback executions in the last minute

MeanRate

The mean rate (events/s) of callback executions

Max

The maximum rate (events/s) of callback executions

Min

The minimum rate (events/s) of callback executions