MQTT Security Fundamentals – Securing MQTT Systems
Welcome to the tenth part of the MQTT Security Fundamentals Blog post series. In the last posts we focused on how to secure MQTT on a protocol level and shared best practices how to implement security on the application level. Todays blog post will focus on the secure deployment of a MQTT system. We will take a look on different layers of security and how to harden the deployment to prevent and mitigate attacks.
There are different layers of security we need to discuss, in particular we are going to look at these layers:
- The Infrastructure
- The operating system
- The MQTT Broker
MQTT brokers are typically deployed on some kind of network infrastructure, so it’s very important to understand the network topology of the target system and it’s important to lock out attackers as soon as possible in order to prevent damage and downtime on downstream systems.
Every connection to a MQTT broker should at least pass one firewall which implements sophisticated rules for accessing downstream components and parts of the infrastructure. If your are able to block attackers at the firewall level, they won’t be able to access any other systems. Unfortunately there is no silver bullet and firewall rules need to be configured according to the concrete use. There are many commercial and open source firewall solutions available.
A rule of thumb is, that only expected traffic gets forwarded to downstream systems. That means, any traffic which you don’t expect in your downstream applications should be blocked. In case of operating an MQTT broker the following traffic could be worth blocking:
- UDP: MQTT uses TCP, so you can block all UDP datagram packets.
- ICMP: While it may not be the smartest idea to block all ICMP traffic, ping and traceroute ICMP packets could be worth investigating as candidates to block.
It’s also a good idea to block traffic to any ports which are not needed for your MQTT system. The following MQTT standard ports should not get blocked by your firewall:
- 1883: This is the default MQTT port. 1883 is defined at IANA as MQTT over TCP.
- 8883: This is the default MQTT port for MQTT over TLS. It’s registered at IANA for Secure MQTT.
If you are in full control of your MQTT system and know the IPs of your MQTT clients, you should only allow traffic by the defined IP ranges. This will lock out any clients which are not in the defined IP ranges.
Load balancers are often used to distribute MQTT traffic to different MQTT brokers, e.g. if the brokers are operated in a cluster. Often these load balancers are proxying the traffic, so it’s important to have a load balancer deployed which can handle the traffic and the connection amount you’re expecting for your production system. Most of the commercial hardware based load balancers can handle huge amounts of connections and are designed for high-traffic scenarios, so typically these just work out of the box.
Load balancers typically don’t add much additional security but they are very useful to prevent overload of downstream systems and distribute the traffic to multiple MQTT brokers. Most load balancers are also able to throttle traffic if the traffic is unusually high and you need to slow down MQTT clients in order to prevent overloading of MQTT brokers.
A DMZ is a demilitarized zone and typically internet facing services (like a MQTT broker) reside here. A DMZ is a subnetwork and all access to downstream services (like databases, internal application services) is protected by an additional firewall. So if an attacker gets access to one of your services in the DMZ the attacker doesn’t get access to other systems (if the firewall is configured properly). In case of a dual-firewall scenario, often firewalls from different vendors are used so attackers can’t use the same compromising techniques if they are able to exploit security holes of a deployed firewall. The Wikipedia article about DMZs is worth a read if you’re not familiar with the concept.
If you are using HiveMQ plugins to access any internal business services (e.g. webservices or authentication services), we strongly recommend deploying a DMZ.
MQTT Brokers are most likely installed on servers (virtual and physical) and these servers run an operating system. Before talking about MQTT broker application security, it’s very important to understand that many attacks focus on security holes in the operating system or software and services which are typically installed with the operating system. While it’s out of scope of this blog post to give a complete overview of hardening servers, we want to share some tips we found useful for making servers more secure.
Please note that the following tips are only valid for Linux based servers.
Keep libraries and software updated
There is no bug-free software and unfortunately some bugs result in security holes. So please keep your system and your software up-to-date. Recent famous security related bugs like the GHOST vulnerability were very simple to fix: By updating specific libraries and software (in the GHOST case, the glibc version). If you are relying on libraries which implement cryptographic algorithms like openSSH or openSSL, it’s critical to update these if new security holes are revealed, because these insecure libraries can affect other software. Fortunately it’s very easy to stay up-to-date since most Linux distributions have a dedicated package manager which can update all outdated software at once.
Disallow root access and use SSH keys for SSH.
If you are using SSH to connect to your server, always disallow root access via SSH, especially if your SSH port is open to the world. Another best practice is to disable password authentication via SSH and use SSH keys for authentication. This adds another layer of security because brute force attempts won’t succeed (in contrast to weak passwords if using password authentication).
Fail2Ban is a neat software which scans different log files on your linux box (like the SSH log) and detects brute force attacks. It will automatically update your firewall (like iptables) in order to lock out the attackers. This tool is invaluable if access to SSH ports is possible via the internet.
Set up iptables
If you don’t have any external firewall deployed, setting up iptables should be mandatory. Iptables is a software firewall which is preinstalled on most Linux distributions. Even if you have external firewalls, using iptables is highly recommended. Some people may find configuring iptables challenging, fortunately there are tools available which eases the burden of maintaining complex iptables rules, like ShoreWall or UFW. Other tools like Fail2Ban rely on iptables and automatically update the rules if malicious clients are detected.
Security Enhanced Linux is a kernel module which provides a mechanism to enforce security policies for access control. Having SELinux configured properly can significantly improve the overall security of the system. It has the reputation of being hard to understand and maintain, though. The time to learn and understand SELinux is well invested, since a well configured SELinux installation can prevent attackers from doing malicious things, even if they gained access to your system. It also prevents applications from behaving differently after updating, so even if an attacker was successful in replacing a software with a malicious counterpart (or tricked you into installing that software), it won’t do too much harm if the SELinux policies were good enough.
Hardening Linux systems is a vast topic and we just scratched the surface. Good practical starting points for securing your Linux systems are the following links:
Even if a reliable and secure network infrastructure is in place and your Linux system is hardened, there are still some improvements possible for securing your MQTT broker deployment.
Authentication and Authorization
We talked about the importance of Authentication and Authorization in previous blog posts. These concepts and their implementation are important to make sure that only trusted MQTT clients can connect and don’t interfere with each other, e.g. by separating topic namespaces. X509 client certificates add another layer of security and you should consider using them if it’s feasible for your concrete use case.
Secure MQTT deployments typically use TLS for transport layer encryption, so no eavesdropper can read and intercept the MQTT traffic from broker to clients and vice versa. While TLS adds extra bandwidth overhead to the communication, the gained security is almost always worth it. If you haven’t read it yet, we suggest to take a look at our blog post about MQTT and TLS.
If you know the bandwidth usage characteristics of your MQTT deployment beforehand, throttling MQTT clients can add additional protection for overloading MQTT brokers. Today bandwidth is still expensive and limited. Few malicious MQTT clients with enough bandwidth available can overload your system very quickly and consume all your bandwidth which may lead to service degradation and can be very expensive.
HiveMQ allows to throttle on a global and a per-client basis. You can limit the total incoming bytes per second and the total outgoing bytes per second independently, so if you know you can only afford 20mbit of traffic (otherwise traffic would get too expensive) but your network and network interfaces can handle 100mbit, it may be wise to throttle the broker to this limit. These limits can be changed while HiveMQ is up and running and no restart is needed.
In addition to the global throttling, it’s possible to throttle specific clients (e.g. based on client identifier) with the HiveMQ plugin system after the clients authenticated successfully.
MQTT defines a maximum message size of 256MB. In most MQTT deployment scenarios messages are much smaller, often small than a kilobyte. If you know your usage scenario very well and you know the maximum message size that can occur, it’s a wise decision to decrease the maximum allowed message size to that limit. Malicious MQTT clients could send huge messages otherwise, which may result in excessive memory consumption and unneeded bandwidth usage.
HiveMQ allows to limit the maximum message size on a global and per-client basis.
We have seen that securing MQTT systems – like any IT system – can be challenging and security needs to be considered at many levels. We have seen that it’s important to know your network infrastructure and your servers very well and they need to be configured conscientiously. If you missed an important tip or feel that we should add additional things, let us know in the comments!
So that’s the end of part ten in our MQTT Security Fundamentals series. If you have any comments or suggestions, we would love to hear your feedback in the comments below!
If you want to be notified as soon as the next part is released, simply sign up for our newsletter below. This brings you fresh content about MQTT and HiveMQ once a week. If you prefer RSS, you can subscribe to our RSS feed here.