Running a HiveMQ cluster with Docker and Kubernetes
The use of containerization and orchestration mechanisms like Docker and Kubernetes is ever growing among all IT deployments. Both the invention and spreading of the DevOps principle as well as the increasing market share and importance of cloud computing providers like Amazon Web Services or Google Cloud Platform are two of the main contributing factors behind this trend. This blog post shows that with the use of the HiveMQ DNS discovery Docker image it is simple and convenient to deploy a HiveMQ cluster with Docker and Kubernetes.
Docker and Kubernetes
Docker is a wide spread software project, providing software containers that aim to make application deployments more easy and portable. Kubernetes is an open-source container-orchestration system for automating deployment, scaling and management of containerized applications. It was originally designed by Google and is now maintained by the Cloud Native Computing Foundation. It aims to provide a “platform for automating deployment, scaling, and operations of application containers across clusters of hosts”. It works with a range of container tools, including Docker.
DNS service discovery
DNS service discovery (DNS-SD) is a way of using standard DNS programming interfaces, servers and packet formats to browse the network for services. The HiveMQ DNS discovery plugin utilizes this technology to generate a discovery method for HiveMQ cluster nodes that is both very dynamic, which means that arbitrary numbers of HiveMQ broker nodes can be added and removed at runtime, and low maintenance. As long as a DNS discovery service, providing Round-robin A records is running and configured correctly, the configuration efforts on the broker side are very little and details about the logical network configuration are inconsequential.
Round-robin in this case means that the DNS service responds to a request for a service with a list of all available hosts (or pods in Kubernetes) for this service.
A records are DNS records that point to an IP address directly.
The HiveMQ DNS Discovery Plugin will utilizes these A records for cluster discovery.
DNS-SD is a perfect candidate for the use with orchestrated containerized environments or most cloud providers.
Installation and configuration
To achieve a dynamic HiveMQ cluster with Kubernetes, which uses the Kubernetes DNS-SD for cluster discovery, the following is required:
- HiveMQ Docker image
- HiveMQ DNS discovery plugin
- Kubernetes cluster environment
HiveMQ Docker image with DNS cluster discovery
A Kubernetes cluster requires the existence of containerized services, to build a cluster. In this example we will be using the HiveMQ DNS Docker image from Dockerhub, which includes the HiveMQ DNS Discovery Plugin.
If you want to create your own Docker images, please take a look at our HiveMQ Docker Images Repository.
Creating a Kubernetes headless cluster service
To enable the utilization of DNS-SD in a Kubernetes deployment, a headless service providing Round-robin A records is necessary.
A headless Kubernetes service enables direct reaching of each individual pot. It is accomplished by setting clusterIP:none in the services specs.
By creating this headless service without selectors for the pod, we also ensure that the DNS service of Kubernetes will return Round-robin A records.
An appropriate YAML configuration file for HiveMQ Kubernetes cluster looks like this:
Line 17: We are using the official HiveMQ 3 DNS discovery image from Dockerhub
Line 27: Make sure to set the value for HIVEMQ_DNS_DISCOVERY_ADDRESS according to your Kubernetes namespace and configured domain.
Download this file.
Line 58: We set clusterIP: None, making the hivemq-cluster1 service headless
Go to the Kubernetes Web UI, click on “+CREATE”, “Upload YAML or JSON file” and select the hivemq-k8s.yaml file and press “Upload”.
As soon as Kubernetes is finished with building the environment we have a working 3 node HiveMQ cluster.
To verify the that the cluster discovery is working properly go to hivemq-replica replication controller and check the log file for one of the pods.
A correct configuration will result in a log line reading
... INFO - Cluster size = 3, members ...
This is a sure sign that the cluster discovery is working properly.
Creating necessary load balancers
The next and last step to our functional HiveMQ cluster is creating 2 separate load balancers for the MQTT (1883) and Web UI (8080) port, so we enable external access to the HiveMQ cluster nodes.
For the MQTT load balancer use the following YAML configuration file and create the service the same as we did the HiveMQ replication controller.
Go to Services -> hivemq-mqtt to find the external end point of the load balancer that can be used to establish MQTT connections to our HiveMQ broker cluster.
The load balancer for the HiveMQ Web UI required the use of sticky sessions. Otherwise the login session information gets lost when browsing to the Web UI.
Make sure your individual Kubernetes environment allows the use of sticky sessions
The following YAML configuration file can be used to leverage the connecting client’s IP for sticky sessions.
You can access the HiveMQ Web UI via this load balancer’s external endpoint with your web browser of choice.
That’s it! We now have fully functional, accessible HiveMQ cluster inside our Kubernetes environment, which we can dynamically scale with the help of our replication controller.
Adding a HiveMQ license to your Docker image
If you are a HiveMQ customer and want to use your license with Kubernetes and HiveMQ DNS Discovery Docker image you can base64-encode the license and set the resulting String as the environment variable
Summary and resource list
Choosing the example of Kubernetes this blog post showed that the new HiveMQ Docker image, including the HiveMQ DNS discovery plugin can be leveraged to create dynamic, auto scaling HiveMQ cluster. Likewise the HiveMQ DNS cluster plugin can be use with any other environment that supports DNS-SD like AWS and Route53 or Google Cloud DNS. If you’re looking to utilize the HiveMQ DNS cluster discovery in your specific environment and are facing difficulties, please do not hesitate to leave us a comment or contact us at: email@example.com
Here are some useful resources: