Top 5 Best Practices When Scalability Testing Your MQTT Broker

In one of our blogs, Top 10 IoT Scalability Tests for an MQTT broker we outlined scalability tests that can help you make sure that your IoT application fulfils your requirements in terms of messaging performance and reliability. These tests are a necessary tool to create peace of mind, especially when you launch a product that has a large potential to scale. Proper scalability testing can also vastly improve resource and capacity planning.

Today’s post will highlight the best practices we have developed when designing and ultimately running the scalability tests for your MQTT broker and demonstrate best practices to achieve your desired outcomes.

1. System Testing: Test Your MQTT Broker in the Context of the Total System

Modern IoT applications are often based on an architecture that includes a multitude of different systems and services. An MQTT broker typically serves as the central, critical communication hub between IoT devices and all surrounding services and systems in the cloud and the backend.

In our last blog post, we discussed that it is vital to test the MQTT brokers performance in a holistic manner. One crucial testing point is the maximum message throughput. It is true that clarity about the MQTT broker’s capability to handle a certain number of requests is helpful. In reality, there are a number of other systems that could act as a limiting factor or bottleneck for that number of requests the application can handle in production. Examples for systems that may act as such bottlenecks are Authentication and Authorization systems or SQL databases.

2. Test Under Real World Conditions

Large scale IoT applications that take advantage of a high performance distributed MQTT broker cluster are handling hundreds of thousands or even millions of concurrently connected devices that all represent an individual TCP connection between device and broker. Managing those vast amounts of TCP connections is a big challenge for the entire system. When setting up a scale test it is important to create a testing environment that acts as close to the final production setup of the application as possible.

This means making sure to use tools for testing that can create actual MQTT clients, using actual MQTT sessions on the scale of the desired scalability test. Tools that simulate multiple MQTT clients over a single connection can lead to false results when it comes to capacity planning and performance tests.

Similarly, it is pertinent to create a networking infrastructure for the test plan that is comparable to the conditions the live IoT application will be deployed under. Consider the use of multiple networking zones as well as the inclusion of additional network components like load balancers, if those are to be used in production.

Lastly we recommend investing the time to create test scenarios that are as identical to the implemented and expected use cases of the application in terms of basic and advanced MQTT characteristics as possible.

Here are some of the specifics that need to be considered:

• Total number of topics used in

• Number of subscribers per topic

• Usage of wildcard subscriptions

• Total number of publishing clients

• Average rate of publishes per client

• Quality of Service Level used

• Average payload size

• Usage of retained messages

• Usage of the WILL feature

The more realistic these specifics can be set in the scalability tests, the more reliant and meaningful results will be acquired.

3. Define Useful Success Metrics

The best scalability test is only as good as the quality of performance indicators used to measure success. It is pertinent to test the characteristics of the MQTT broker like performance, scalability, reliability or security during the test plan. A common issue is that a scalability test measures the performance of a testing tool or the capabilities of network or infrastructure components like load balancers, instead of testing the MQTT broker performance. MQTT broker solutions that provides a holistic view into its inner workings and service health by exposing a complete set of metrics that can be monitored. A holistic set of metrics is useful for scalability testing as well as application development and operations.

It is equally important to define sensible and proper metrics of success. Depending on the specific use case these metrics can, for example, be:

• Latency: Average message delivery time.

• Capacity planning: The numbers of concurrently connected clients, simultaneous subscriptions, messages per second, connections rates etc. the broker can handle with a given set of resources.

• Availability: Resiliency of the system during unexpected failures and ability to recover from them.

• Reliability: Handing of extreme loads with minimal service degradation.

Different MQTT brokers will perform differently based on the success metrics chosen. This is normal based on a number of principle differences that exist between various MQTT brokers and services.

Some of the differences between general MQTT broker concepts and implementations are:

• Robust MQTT broker implementations offer internal overload protection features, which are extremely useful to increase reliability in production. These mechanisms can have a negative impact on performance tests. Especially in scenarios that use low numbers of high throughput producing clients.

• High performance MQTT brokers are working as distributed system in a cluster and can be optimized for throughput and not latency, leaving room for some spikes in message delivery times compared to single node, single threaded applications that are not capable of handling large amounts of messages at all and do not offer high availability.

• MQTT brokers that are purely optimized for performance implement in-memory persistence only. This comes with a distinct draw back in terms of recovery capabilities as all data is lost as soon as the process stops running.

Precisely defined success metrics can help with designing and evaluating scalability tests.

4. Security First: Test With End-to-End Security Enabled

IoT applications typically deal with sensitive or private data. It is crucial to have this in mind when designing an IoT application. By adhering to IoT security best practices data breaches and the bad press that goes along with them can be avoided. On the application layer of an MQTT broker the best practices are to utilize end-to-end encryption with TLS certificates, an additional device authentication mechanism, and resource separation via user authorization.

Enterprise grade MQTT broker solutions offer ways to extend the broker with Authentication and Authorization based on virtually any third party data system that needs to be integrated.

It is vital to do all your scalability tests for the broker with all security mechanisms activated.

5. Test for Failures

It is important and useful to test the maximum performance an MQTT broker cluster can achieve when everything runs as expected. The results of such tests can help with capacity planning and provide clarity in terms of the general achievability of a desired scale for an IoT application. Reality dictates that it is also essential to test for certain failure scenarios that will inevitably occur. We listed a number of failure scenarios that should be included in any testing plan in our last blog post. Defining desired outcomes for failure tests is crucial.

MQTT brokers should be able to handle network or hardware failure with no or minimum data loss as well allow for zero downtime upgrades.

Summary

• Make sure to test your entire system. Testing isolated applications in a vacuum can be misleading.

• Use tools that allow testing with conditions close to the desired production use case. Avoid unexpected surprises by testing under real life conditions.

• It is important to have clear, defined performance indicators. Make sure that benchmarks are not read wrong by overlooking reliability features.

• Do not overlook security! Introduce the same security measures you want in production in your scalability test.

• Stray from the happy path! Include reliability and recovery test for your MQTT broker that show how it can handle failures and unwanted circumstances.

Conclusion

Scalability testing your IoT application is vital and helpful. Following the best practices in this article and the key scenarios to test, we identified in our last post on the subject, ensures that testing for the central communication component - the MQTT broker - is done in a holistic manner.

We strongly recommend following these best practices for any MQTT broker scalability test. Feel free to contact us directly at contact@hivemq.com, if you have any follow up questions about scalability testing of your IoT application.