Control Center Analytics

The HiveMQ Control Center offers a wide array of analytics functionality. While we do not recommend substituting an operation ready monitoring set up, these analytics can be helpful to get insights to your HiveMQ deployment without the need for any extra efforts.

Check out our collection of official, off-the-shelf HiveMQ Monitoring Extensions


Dropped Messages

This site enables you to analyse dropped messages. A dropped message is a message which is not published by the broker due to reasons explained below.

The following aspects can be analysed:

Detailed information on the reasons for dropped messages is given at the end of this section.
Complete view over the site
Figure 1. Dropped Messages Overview


Dropped Messages by Reason

The reasons why messages were dropped are visualized on the top of the page.

The left diagram visualizes the count of dropped messages over the last hour grouped by the reason.

Dropped Messages per Reason (stacked)
Figure 2. Dropped Messages per Reason (stacked)

You can select or deselect specific reasons by clicking on them in the legend of the diagram. The diagram is updated every 10 seconds.

The right pie chart visualizes the total count and the percentage of every reason.

Dropped Message Reasons

You can select or deselect specific reasons by clicking on them in the legend of the chart.

Message are dropped because of the following reasons:

Table 1. Reasons for dropping messages
Reason Description

Client Message Queue Full

Client’s message queue was full as another message for that client arrived.

QoS 0 Memory Exceeded

Broker wide memory for queueing Quality of Service 0 messages has been exceeded.

QoS 0 Channel Not Writable

Quality of Service 0 could not be delivered because it was impossible to write to the client socket

Maximum Packet Size Exceeded

Message size was bigger than the maximum size of the receiving client

Extension Prevented

Message deliver got prevented by a Publish Inbound Interceptor

Internal Error

Messages that were dropped because of internal errors.


Dropped Messages by Client

This table contains the first 100 clients per Node that dropped messages in the last four days:

clients dropping messages
Table 2. Message Dropping Clients
Column Descriptions

ClientID

The ID of the Client that dropped messages

First Dropped Message

The timestamp of the first message the Client dropped

Last Dropped Message

The timestamp of the last message the Client dropped

Client Message Queue Full

Client’s message queue was full as another message for that client arrived.

QoS 0 Memory Exceeded

Broker wide memory for queueing Quality of Service 0 messages has been exceeded.

QoS 0 Channel Not Writable

Quality of Service 0 could not be delivered because it was impossible to write to the client socket

Maximum Packet Size Exceeded

Message size was bigger than the maximum size of the receiving client

Extension Prevented

Message deliver got prevented by a Publish Inbound Interceptor

Internal Error

Messages that were dropped because of internal errors.

Each column can be ordered ascending or descending. Filtering is available for ClientID, using a "startsWith" comparison.

By clicking on a Client you can navigate to the Client Detail View. Furthermore you can clear all data stored with the Clear Data - button located on right side above the table. Additionally there is the possibility to refresh the data by pressing the Refresh Data - button


Dropped Messages by Shared Subscription Groups

This table represents information about Shared Groups which dropped messages during the last four days. The maximum number of Shared Groups displayed is limited to 100

Shared Subscription Groups dropping messages
Table 3. Dropped messages by Subscription Groups
Column Descriptions

Shared Group

The ID of the Client that dropped messages

First Dropped Message

The timestamp of the first message the Client dropped

Last Dropped Message

The timestamp of the last message the Client dropped

Client Message Queue Full

Client’s message queue was full as another message for that client arrived.

QoS 0 Memory Exceeded

Broker wide memory for queueing Quality of Service 0 messages has been exceeded.

Internal Error

Messages that were dropped because of internal errors.


Detailed explanation of the reasons

The core function of the HiveMQ broker in any MQTT deployment is delivering messages.
For most use cases this message delivery needs to be as reliable as possible and the dropping of messages is not a desired outcome.
This does not mean that every dropped message automatically represents a problem. The HiveMQ Control Center offers an in-depth look into why messages got dropped and who the messages got dropped for.
The following lists explanations for each possible cause of dropped messages as well as some insight towards interpreting them and leveraging this information for debugging purposes.

Client Message Queue Full

Every client has a queue in which messages for the client. Queueing either occurs when the client uses persistent sessions and is offline or when the client is not consuming messages in the same speed as it is receiving messages. When this queue is full, the newest or oldest messages get dropped, dependent on the queuing strategy.
Observing a high number of this kind of dropped messages may indicate that your clients are consuming messages too slow or that their connection is very unstable.

QoS 0 Memory Exceeded

When messages with Qualify of Service 0 is received the broker puts the message into a global in-memory queue.
In case the size of that QoS=0 queue exceeds a global memory limit, additional incoming QoS=0 messages get dropped.
Observing a high number of this kind of dropped messages may indicate that your clients are consuming messages too slow or that their connection is very unstable.

QoS 0 Channel Not Writable

When a Messages with QoS 0 is received by the broker and the socket of the intended receiving client could not be written on, the message is dropped with this reason.
This is caused by a client not being able to read fast enough to keep up with the incoming messages.

Maximum Packet Size Exceeded

MQTT 5 allows the a client side setting for maximum packet size of messages received. When a message is larger than its receiving clients maximum size setting it get dropped with this message.
A large number of this kind of dropped messages indicates that the coordination between sending and receiving clients is not optimal.

Extension Prevented

The Publish Inbound Interceptor allows users to prevent onward delivery of messages. In this case the broker drops the message with this reason
When best practises for extension development are upheld the reason for the delivery prevention will be logged and or forwarded to the client.

Internal Error

If an internal error occurs during handling a message, the message gets dropped. The reason for the internal error is not specified.
Messages dropped for this reason are a clear warning sign for some underlying unexpected problem. Check your HiveMQ log for the referenced Error and contact the HiveMQ Support, if you observe this.


Cluster metrics

This section is designed provide a quick overview and some context on the available cluster metrics of the HiveMQ Control Center dashboard.

The following metrics are displayed on the dashboard:

Cluster total numbers overview
Figure 3. cluster metrics
Table 4. Metrics
Metric Description

Connections

Current amount of active connections on all nodes

Inbound Publish Rate

Current amount of incoming Publishes per second over all cluster nodes

Outbound Publish Rate

Current amount of outgoing Publishes per second over all cluster nodes

Subscriptions

Current amount of Subscriptions and replicas stored in the cluster

Retained Messages

Current amount of Retained Messages and replicas stored in the cluster

Queued Messages

Current amount of Queued Messages and replicas stored in the cluster (may show Queued Messages of already disconnected clean session clients)

Cluster Nodes

Current amount of Cluster Nodes

For more information on clustering, please visit our clustering benefits blog post.


Detailed Metric View

Each of the available metrics is clickable. Clicking on one of the metrics will provide a more detailed visualization of the metric, which is split by node.

Connections

It is possible to choose any metric of the top section like inbound publish rate:

Inbound Publish Rate

The chosen metric is underlined in the top section.

You can toggle the graphs of individual nodes by clicking the respective buttons on the right legend of the graph.

All Nodes are displayed
Figure 4. All nodes selected:
Only one Node is displayed
Figure 5. One node selected:
If you compare both diagrams you will notice that the scale changes automatically.

Trace Recordings

This is a HiveMQ Enterprise Edition feature. Find out more about HiveMQ Editions.

Usually a MQTT broker sends and receives millions of MQTT messages during a small period of time. To find out what is really happening with a client or on a topic by analysing all information from the broker is very time-consuming. For an easier approach in doing this, HiveMQ provides the ability to write log files for clients and topics.

A Trace Recording is a combination of filters which allows you to select messages of specific clients or topics, which are logged to a file in a human readable format. Each trace recording creates a .trace file in your HiveMQ log/recordings folder. Trace recordings are created cluster-wide and every HiveMQ cluster node writes its own trace files in its own log folder.

You can log any MQTT message sent or received by the broker with a filter you apply. A Filter can either be a client identifier or a topic. Many filter combinations are possible. All filters are regular expressions.

To add a trace recording click the Add New button in the right upper corner. More information on adding a trace is given at the <<trace-recording-new, Add new Trace Recording part.

The trace recording log files are stored as trace-recording-name.trace in hivemq/log/recordings.
Trace Site View
Figure 6. Trace Recordings Site

There are four attributes, which define a trace recording explained in the table below.

Table 5. information_active_traces
Attribute Description

State

Running Running, Waiting Waiting, Stopped Stopped or Aborted Aborted

Name

The Name of the trace recording log file

Start

The start date, when HiveMQ begins to record messages for a trace recording

End

The end date, when HiveMQ stops to record messages for a trace recording

See Active Trace Recordings

This is a HiveMQ Enterprise Edition feature. Find out more about HiveMQ Editions.

Active trace recordings are all trace recordings in the state 'Running' Running or 'Waiting' Waiting

Active trace recordings can be viewed, downloaded, stopped and deleted.

View a trace recording by clicking on its name. For more detail, have a look here: Trace Recording Details
Download a trace recording by clicking ⇒ Download. For more detail, have a look here: Download
Stop a trace recording by clicking ⇒ Stop

A stopped trace recording is still available in your HiveMQ log/recordings folder and in the Finished Trace Recordings table, but cannot be started again.

Delete a trace recording by clicking ⇒ Delete

A deleted trace recording is not visible anymore. All its log files are deleted from disk.
Active Trace Recordings
Figure 7. Active trace recordings table

Add new Trace Recording

This is a HiveMQ Enterprise Edition feature. Find out more about HiveMQ Editions.

To add a new trace recording to your HiveMQ click the Add New button in the right upper corner of the Trace Recordings page. The following information needs to be specified Basic Settings, Filters and MQTT Packets

At the top right corner is either a Start Trace button or a Schedule Trace button.
When the start date is in the future, the trace recording will be scheduled. When the start date is now or in the past, the trace recording is started instantly.

Site for adding a new Trace
Figure 8. Site for adding a new Trace

Trace Recording Details

This is a HiveMQ Enterprise Edition feature. Find out more about HiveMQ Editions.

If you clicked on the name of a trace recording, you will be redirected to 'Trace Recording Details', where you can see the sections 'Basic Settings', 'Filters' and 'Trace Recording Contents'. All information is read only.
For more detail about the sections see Add new Trace Recording

Stop a trace recording by clicking ⇒ Stop
Download a trace recording by clicking ⇒ Download. For more detail, have a look here: Download

A stopped trace recording is still available in your HiveMQ log/recordings folder and in the Finished Trace Recordings table, but cannot be started again.
Trace Recording Details

Basic Settings

This is a HiveMQ Enterprise Edition feature. Find out more about HiveMQ Editions.

All fields in 'Basic Settings' are required. If any field failed to validate, you get a red warning message below the corresponding input field.

Choosing the basic settings

Name
The name defines the filename of the .trace log file in your HiveMQ log/recordings folder. It must be unique, contain at least three characters and only combinations of numbers, letters, dashes and underscores are allowed. E.g. 'publishing-clients', 'subscribe_my_topic_1' etc.

Start
When the start date is now or in the past, HiveMQ will start recording log messages immediately after successfully creation of the trace recording.
When the start date is in the future, HiveMQ will schedule the trace recording. The default value is now.

It must be before the end date and must have this format: yyyy-MM-dd HH:mm:ss
E.g. 2017-10-18 19:42:00

If the start date is after the end date, you can either change the end date or the start date to make the date fields valid.

You can type date and time into the input field or click on the calender icon to choose date and time per mouse. See below.

date time chooser

End
The end date determines when HiveMQ stops the trace recording. The default value is now plus one hour.

The end date must be in the future and after start date. It must have this format: yyyy-MM-dd HH:mm:ss
E.g. 2017-10-18 20:42:00

If the end date is before start date, you can either change the end date or the start date to make the date fields valid.

You can type date and time into the input field or click on the calender icon to choose date and time per mouse. See below.

date time chooser

Filters

This is a HiveMQ Enterprise Edition feature. Find out more about HiveMQ Editions.

At the filters section, you can add Client Identifier and Topic filters to the trace recording. At least one filter (ClientID or Topic) must be set. Filters are based on regular expressions.

For more information about regular expressions please look here.
Choosing the filters

You can set and combine as many ClientID and Topic filters as you want. Contents are recorded, if at least one filter of each category matches. For example, if you set two ClientID filters 'client1' and 'client2' and one Topic filter 'my/topic' all chosen contents will be recorded for 'client1' and 'my/topic' as well as for 'client2' and 'my/topic'.

example filters
Table 6. Possible Trace Recording Filters
Filter Instructions

Client Identifier:

To add a Client Identifier filter, just type a regular expression into the field below and then press Enter or click the Add ClientID Filter Button

Topic:

To add a Topic filter, just type a regular expression into the field below and then press Enter or click the Add Topic Filter Button

To remove a filter click on "x" of the filter you want to remove.

MQTT Packet Selection

This is a HiveMQ Enterprise Edition feature. Find out more about HiveMQ Editions.

In this section you can choose what kind of MQTT messages should be traced. MQTT Messages are messages sent or received by MQTT clients and MQTT brokers You can select the following MQTT Message types to be logged. At least one message type must be selected:

Client filters can be used to filter any kind of MQTT message, while topic filters apply only for MQTT messages which contain a topic. These are PUBLISH, SUBSCRIBE and UNSUBSCRIBE.
Choosing the trace recording contents
Table 7. Possible MQTT Messages
Content Description

CONNECT

A CONNECT message is sent to the broker, when a client connects to a broker

CONNACK

A CONNACK message is sent to the client, when a broker acknowledges the connection of a client

SUBSCRIBE

A SUBSCRIBE message is sent to the broker, when a client subscribes to a topic

SUBACK

A SUBACK message is sent to the client, when a broker acknowledges the subscription of a client

PUBLISH

A PUBLISH Message is sent to the broker, when a client publishes to a topic

PUBACK

A PUBACK message is sent to the client, when a broker receives a PUBLISH with QoS 1

PUBREC

If a broker gets a QoS 2 PUBLISH it will process the publish message accordingly and acknowledge it to the sender with a PUBREC message

PUBREL

If the client receives the PUBREC it can safely discard the initial publish, because it knows that the counter part has successfully received the message. It will store the PUBREC and respond with a PUBREL Message

PUBCOMP

If the broker gets the PUBREL it can discard every stored state and answer with a PUBCOMP. The same is true when the sender receives the PUBCOMP.

UNSUBSCRIBE

An UNSUBSCRIBE message is sent to the broker, when a client wants to remove a subscription from a topic

UNSUBACK

A broker will acknowledge the request to unsubscribe with the UNSUBACK message.

PINGREQ

A PINGREQ is sent by the client and indicates to the broker that the client is still alive.

PINGRESP

When receiving a PINGREQ the broker must reply with a PINGRESP packet to indicate its availability to the client.

DISCONNECT

A DISCONNECT message is sent to the broker, when a client disconnects from a broker

To learn more about the MQTT Message types, have a look at our MQTT Essentials

A sample configuration might look like:

Chosen trace recording contents

Finish Trace Recording Creation

This is a HiveMQ Enterprise Edition feature. Find out more about HiveMQ Editions.

After you configured the trace recording you can press the "Start/Schedule Trace Recording" button in the top right corner. In case you configured something wrong, a notification with the invalid settings is displayed:

invalid trace settings

If everything is configured correctly, a confirmation with a duration will be displayed. You can stop the process by clicking "Cancel". To confirm the trace recording press "Start/Schedule" and the trace recording will be added to HiveMQ and you will be redirected to Trace Recordings.

The trace recording log file is located in HiveMQs log/recording folder.
Confirm
Regular Expressions

Regular Expressions or short regex are search patterns to find any matching character sequences.

For example if you want to find all names with ik in it, the pattern would be simple ik.

If you want to find all words with a a followed by at least one n, the regex would be an+. The + works as quanitification. It defines that at least one n must follow the a. This regex would find sequences like an, ann , annn, but not a, n, na.

These quantifiers always refer to the expression before:

  1. a+: + refers to a

  2. `: ` refers to [a-c]

There are more quantifiers:

Table 8. quantifiers
Quantifier Description Example matches matches not

?

one or zero times

ha?llo

hallo hllo

haallo

*

zero or more times

ha*llo

hllo, `hallo, haallo, …​.

hakallo

+

one or more times

ha+llo

hallo, haallo, haaallo

hllo

{number}

exactly number-times

hal{2}o

hallo

halo, hao, halllo`

{min, }

at least min times

hal{2,}o

hallo, halllo

halo

{min, max}

at least min and maximal max times

hal{1,2}

halo, hallo

hao, halllo

With the brackets [ ] a set of characters can be defined that matches a single character of this set. For example h[a, e]llo matches hallo and hello. It is possible to set ranges to: [c-f] would match c, d, e and f.

Further it is possible to use the wildcard-character: . to match any character. For example h.llo matches hallo, hbllo, h llo, and so on.

Combined with the quantification * any string can be matched with .* .

As many characters are used as metacharacters they need to be escaped with a leading \ to be used as character:

Table 9. metacharacters
Metacharacter Escaped

*

\*

+

\+

.

\.

[ and ]

\[ and \]

{ and }

\{ and \}

( and )

\( and \)

\

\\

|

\|

Finished Trace Recordings

This is a HiveMQ Enterprise Edition feature. Find out more about HiveMQ Editions.

Finished trace recordings are all trace recordings in the state 'Stopped' Stopped or 'Aborted' Aborted

Finished trace recordings can be viewed and deleted.

View a trace recording by clicking on its name. For more detail, have a look here: Trace Recording Details
Download a trace recording by clicking ⇒ Download. For more detail, have a look here: Download
Delete a trace recording by clicking ⇒ Delete

A deleted trace recording is not visible anymore. All its log files are deleted from disk.
Finished traces table

Download finished trace recordings

This is a HiveMQ Enterprise Edition feature. Find out more about HiveMQ Editions.

Every running, stopped and aborted trace recording can be downloaded as a zip file. This zip file contains a folder per node with every requested trace recording file.
To download a trace recording click ⇒ Add New

The download will then be prepared in a few steps:

  • When the trace recording is in running state, all trace files will be copied into a temporary folder.

Copying Files
  • All trace recording files from every cluster node will be requested and temporarily stored. ( Progress from 0% - 50% )

Requesting Files
  • A zip file will be created. ( Progress from 50% - 100% )

Compressing Files
  • Temporary files will be deleted. ( Progress not visible )

Done

During every step you can cancel the download by clicking the cancel button. When the download is ready you can save the zip file to your local file system or just close the window.

If something went wrong during the preparation, a warning message will be displayed.

Possible warnings are:

  • Not enough disk space (At least 100 MB must be free on every cluster node)

DiskSpace
  • Files not found

NotFound
  • Can’t read files

CantRead
  • Can’t write files

CantWrite
  • Feature disabled during rolling upgrade

Disabled
  • Any internal error

Internal

Some warnings may have a link to Dashboard with the cluster node as parameter to select the affecting nodes statistic tab.