SYS-Topics: Why you shouldn’t use SYS-Topics for Monitoring
Written by The HiveMQ Team
Published: March 29, 2016
Monitoring your systems is extremely important for production environments. When deploying a MQTT broker or a cluster of MQTT brokers, the operation team needs to know the health of the MQTT servers and uses monitoring data to circumvent foreseeable future problems. In the past few years a handy feature of many MQTT brokers, the so called SYS-Topics, gained more and more popularity not only for debugging and developing MQTT systems but also for monitoring. This blog post discusses the use of SYS-Topics for monitoring and you will learn when to use - and more importantly - when NOT to use SYS-Topics.
Many MQTT brokers, including HiveMQ, implement SYS-Topics. These topics are special meta topics which are used by the broker to publish information about the broker itself and its MQTT client sessions. All SYS-Topics start with $SYS and are read-only for MQTT clients, so publishing to these topics is prohibited. SYS-Topics are mentioned in the MQTT 3.1.1 specification (4.7.2) but are a non-standard feature that are available in some MQTT brokers like HiveMQ or mosquitto.
An unofficial SYS-Topic specification is available on the MQTT Wiki, it is however subject to change and applications should not rely on these information.
Brokers publish these SYS-Topics on a periodical basis, the default on most brokers is 60 seconds. All MQTT clients that subscribe to one or more SYS-Topics receive the current value on the SYS topics as soon as they subscribe. After the subscription was successful, the client will receive metrics on periodical basis. Some static SYS-Topics (e.g. broker version) are only published upon subscription.
Debugging and Development with SYS-Topics
SYS-Topics are fantastic for developing and debugging MQTT applications. Developers can quickly monitor the current state of the broker and calculate and verify metrics - like message amplification rate or network metrics. Applications like MQTT.fx even provide native support for SYS-Topics embedded in a beautiful interface.
As you can see in the following video, you can get a quick overview of the MQTT brokers metrics by subscribing to the SYS-Topics:
SYS-Topics for Production Monitoring
The SYS-Topics definitely have their use-cases for development and debugging. However, SYS-Topics are not suitable for monitoring production broker instances. If you need to rely on the availability guarantees of your MQTT broker, consider the use of a real monitoring system, SYS-Topics do not replace monitoring applications.
SYS-Topics are not suitable for production monitoring due to the following 5 reasons:
1. Metric resolution is not good enough
The metric publishing interval for most MQTT brokers is 60 seconds by default. Monitoring interfaces like JMX allow to poll the most current data whenever needed. So the broker does not dictate the monitoring system how the resolution should be. The monitoring system can decide how often new data should be collected, without modifying any configuration on the MQTT broker.
2. SYS-Topics provide only a subset of available metrics
While the metrics provided by SYS-Topics can be useful, they are only a small subset of all available metrics. HiveMQ exposes more than 100 different metrics (the number of exposed metrics also increase with every release). The SYS-Topic Metrics usually are more focused on MQTT session monitoring and thus don’t provide metrics that can give you an overview of the brokers health at a quick glance. The most interesting metrics for operation teams are usually product dependant, so it’s unlikely that a common SYS-Topic standard will cover these in the future.
3. SYS-Topics expose internal information to potential attackers
It’s a common best practice to conceal the actual software version used in a deployment from attackers to make it harder to use exploits that are targeted towards a specific software version. Often you even want to hide the actual software used from attackers. While this only provides Security by Obscurity (which you should never rely on as only security measure!), it’s still important to hide such deployment information. SYS-Topics expose information like Broker Software Used and Version Number to any subscriber. These may be valuable information for attackers but often don’t provide too much value to actual legitimate subscribers.
4. It’s hard to monitor broker clusters with SYS-Topics
HiveMQ SYS-Topics expose statistics only for the current cluster node. If someone would like to build a custom monitoring solution based on SYS-Topics, MQTT connections to all cluster nodes need to be established. Typically MQTT clusters are behind a load balancer, so often it’s not even possible to connect to a specific cluster node. Monitoring integrations with e.g. Graphite are built with clustering in mind and so it’s easy get a quick health overview of the whole cluster. That’s much harder to do with SYS-Topics.
5. The monitored channel should be separated from the monitoring channel
A good monitoring practice is to use a dedicated monitoring channel instead of using the same channel you are monitoring. With MQTT brokers this means that if you are monitoring the MQTT communication, you shouldn’t use MQTT (SYS-Topics) to monitor the MQTT communication. It’s the same as: Don’t use e-mail alerts for monitoring e-mail servers. If the MQTT communication is not available for any reason, you won’t get any monitoring data. The monitoring data may be most valuable especially when something bad happened. So if you’re relying on these data for alerting you may have a problem. If you’re using a dedicated monitoring channel like JMX, Graphite or Nagios, a problem with the MQTT communication should not affect your monitoring.
How to disable SYS topics for HiveMQ
SYS-Topics are enabled by default when starting HiveMQ. The broker comes with a standard plugin that hooks in the SYS-Topic functionality. If you want to disable the SYS-Topics, just delete the hivemq-sys-topic-plugin-3.x.x.jar file from the plugins folder.
MQTT SYS-Topics are useful for debugging and developing MQTT applications but are not optimal for monitoring MQTT deployments and alerting operation teams in case something bad happened. Dedicated monitoring interfaces like JMX do a better job at monitoring and provide better flexibility.
What are your experiences with SYS-Topics? Do you use them regularly or do you even use them for production monitoring? Let us know in the comments!