Deploy a HiveMQ cluster on Azure Kubernetes Service (AKS) as a StatefulSet with high-performance Azure storage disks

Today’s blog post focuses on deploying a HiveMQ cluster on Microsoft’s Azure Kubernetes Service (AKS) using a StatefulSet that provides built-in support for persistent volumes. To guarantee the disk performance you need for high-performance MQTT use cases, we show you how to specify premium Azure storage disks for your HiveMQ data directory.

Introduction

Kubernetes provides several built-in workload resource types such as Deployment, DaemonSet, and StatefulSet to make installing and maintaining containerized applications on Kubernetes much easier. Each workload resource configures a controller that creates and maintains a set of Kubernetes pods on your behalf so you don’t need to manage the pods directly. Different types of workload resources are ideal for different applications and use cases.

By default, the HiveMQ Kubernetes Operator uses the Deployment workload resource type to install and maintain a HiveMQ cluster. Since the Deployment type lacks built-in support for persistent volumes, ephemeral volumes are used for the HiveMQ data directory. Typically, the ephemeral volumes are mounted from the local storage of a Kubernetes node and are therefore shared between pods on the node. The use of shared ephemeral storage makes it difficult to guarantee the performance (IOPS) for individual pods or applications. Input and output per second IOPSs is the standard way to measure how quickly a given storage device can read and write data. Higher values mean a device is capable of handling more operations per second. For high-performance MQTT use-cases, it is important to exert control on which type of storage is used.

Since HiveMQ 4.7, the HiveMQ Kubernetes Operator can also be used to install HiveMQ clusters with the Kubernetes StatefulSet resource type. This is good news for MQTT use cases that require consistently high disk performance and can benefit from using a separate premium storage volume. For example, solutions that need to provide large numbers of retained messages.

Together with the HiveMQ operator, the StatefulSet controller maintains the deployment and scaling of pods for your HiveMQ cluster to make sure the number and type of pods you specified for your application are always running. Additionally, StatefulSets provide guarantees about the ordering and uniqueness of the pods and include specifications for persistent volumes. PersistentVolumes (PV) are storage volumes in the Kubernetes cluster that are created and managed independently of the lifecycle of a pod. AKS can select from various Azure disk storage offerings to provide persistent storage.

Prerequisites

• Microsoft Azure account
• A running AKS Kubernetes cluster with available compute resources for HiveMQ
• The Kubernetes command-line tool, kubectl, installed locally
• The Azure command-line tool, az, installed locally
• The current version of Helm, installed locally
• HiveMQ license (optional). If you don’t have a valid HiveMQ license, the HiveMQ cluster you deploy automatically uses a trial version that is restricted to non-production use and 25 concurrent connections. To test and evaluate HiveMQ with more connections, contact HiveMQ.

NOTE: This article does not go into the details on how to install an AKS cluster. If you need some help to get started with AKS, see Quickstart: Deploy an Azure Kubernetes Service and HiveMQ with Azure Kubernetes Service.

Step 1: Install the HiveMQ Kubernetes Operator and a HiveMQ cluster on AKS with Helm

In this step, you use the HiveMQ Helm chart to install the HiveMQ Kubernetes Operator and a HiveMQ cluster on AKS with the default Deployment resource type and you define a custom installation file.

• Use kubectl to create a namespace for HiveMQ on your AKS cluster:

  1	kubectl create namespace hivemq

• Add the HiveMQ Helm Chart to your local Helm repository:

  1	helm repo add hivemq https://hivemq.github.io/helm-charts

  NOTE: If you have already installed the HiveMQ Helm Chart, use the following command to ensure that you are running the most current version:

  1	helm repo update

• Create a custom installation file named myCustomValues.yaml with the following values:

  1 2 3 4 5

  hivemq: logLevel: INFO nodeCount: 2 cpu: 4 memory: 4Gi

• Use Helm to install the HiveMQ Kubernetes Operator and a HiveMQ cluster that uses the settings you specified in the myCustomValues.yaml file:

  1	helm install myhivemq hivemq/hivemq-operator -f myCustomValues.yaml

  This command does the following:
• Extends the Kubernetes API with the HiveMQ Cluster Custom Resource Definition (CRD).
• Installs the HiveMQ Kubernetes Operator as a custom controller for HiveMQ clusters.
• Installs a HiveMQ cluster with 2 nodes and the default HiveMQ configuration settings.
  NOTE: This HiveMQ cluster requires 8 CPUs and 8 Gi of memory. If these resources are not available for your cluster, the HiveMQ pods remain in a pending state. For testing purposes, you can reduce the CPU and memory settings of the HiveMQ cluster.
• Saves the instance of the Helm chart with the release name myhivemq.
  NOTE: A message on your command-line interface confirms that the Helm deployment was successful.

• To verify that your HiveMQ cluster is installed, enter:

  1	kubectl get hivemq-clusters

  As soon as the displayed state shows RUNNING, your HiveMQ cluster is ready to go.

• To access the HiveMQ Control Center of the HiveMQ cluster, forward the port of the Control Center service through kube-proxy:

  1 2	kubectl get svc kubectl port-forward svc/hivemq-myhivemq-cc 8080

  To open the HiveMQ control center for your HiveMQ cluster, navigate to http://localhost:8080 and enter the default control center credentials in the HiveMQ control center login dialog (username: admin, password: hivemq). The control center dashboard gives you detailed information for your HiveMQ cluster and MQTT clients. To learn more about the HiveMQ control center, see HiveMQ Control Center.

Step 2: Install a HiveMQ cluster with the StatefulSet template

In this step, you use Helm to delete and reinstall your HiveMQ cluster with the StatefulSet resource type and a PersistentVolume that uses the default Azure StorageClass.

• Delete the previous HiveMQ cluster:

  1	helm delete myhivemq

• Edit your myCustomValues.yaml installation file with the following values:

1
2
3
4
5
6

hivemq:
  logLevel: INFO  
  nodeCount: 2
  cpu: 4
  memory: 4Gi
  controllerTemplate: cluster-stateful-set.yaml

• Use Helm to install a new HiveMQ cluster with the StatefulSet template:

  1	helm install myhivemq hivemq/hivemq-operator -f myCustomValues.yaml

  This command installs your HiveMQ cluster using the StatefulSet template. Since no custom persistent volume settings are specified, the HiveMQ operator installs a default persistent volume with 100Gi storage capacity and the default StorageClass of your AKS cluster.
• To verify that the default persistent volume was successfully created, enter:

  1	kubectl get persistentvolumeclaim

  The result shows the created volume:

  1 2	NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS data-myhivemq-0 Bound pvc-23b03202 100Gi RWO default

Step 3: Install a HiveMQ cluster with Azure managed-premium SSD storage

In the last step of this example, you use Helm to delete and reinstall your HiveMQ cluster so that you can specify a PersistentVolume that uses the Azure premium-managed StorageClass for high IOPS performance.

• Delete the previous HiveMQ cluster:

  1	helm delete myhivemq

• Adjust your myCustomValues.yaml installation file to specify a high-performance SSD-backed disk for your HiveMQ cluster:

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

  hivemq: logLevel: INFO nodeCount: 2 cpu: 4 memory: 4Gi controllerTemplate: cluster-stateful-set.yaml volumeClaimTemplates: - kind: PersistentVolumeClaim apiVersion: v1 metadata: name: data spec: accessModes: - ReadWriteOnce resources: requests: storage: 300Gi volumeMode: Filesystem storageClassName: managed-premium

• Use Helm to install your HiveMQ cluster with the new values:

  1	helm install myhivemq hivemq/hivemq-operator -f myCustomValues.yaml

  This command installs your HiveMQ cluster using the StatefulSet template and a VolumeClaimTemplate that specifies a disk with 1100 IOPS and 300Gi storage capacity that is suitable for production workloads.
  
  Based on the PersistentVolumeClaims (PVC) that you specify in the volumeClaimTemplates section of your myCustomValues.yaml file, the Kubernetes StatefulSet controller provisions a PersistentVolume for each HiveMQ pod. The storageClassName that you specify in the volumeClaimTemplates setting identifies the type of Azure storage disk you want the StatefulSet controller to provide for the persistent volume. StorageClasses map Azure disk resources to Kubernetes. If you do not request a specific StorageClass, the default StorageClass is used.
  Azure provides several StorageClasses that are installed and available for use in every AKS Kubernetes cluster.
  • To view the available Azure StorageClasses on your AKS cluster, enter:

    1	kubectl get storageclasses

    The result of this command lists the StorageClass name and key details for each available Azure storage disk type on the AKS cluster:

    1 2 3 4 5

    NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXP. azurefile kubernetes.io/azure-file Delete Immediate true azurefile-premium kubernetes.io/azure-file Delete Immediate true default (default) kubernetes.io/azure-disk Delete WaitForFirstConsumer true managed-premium kubernetes.io/azure-disk Delete WaitForFirstConsumer true

    
    The default Azure StorageClass maps to the Azure Standard SSD storage offering. Azure Standard SSD storage is designed for medium IOPS workloads of up to 500 IOPS. Although Azure states higher max burst performance for the standard disk, the heightened performance is only available for brief periods. For your HiveMQ cluster on AKS, look for a fast and reasonably-priced disk that provides consistent performance of at least 900 IOPS.
    Azure managed-premium storage offers much more suitable options. Starting at a size of 256Gi, the Azure Premium SSD disks in the managed-premium StorageClass deliver sustained performance of 1100 IOPS. For high-performance HiveMQ workloads, Azure Premium SSD disks are the recommended choice.

• To verify that the persistent volume was successfully created as specified in your myCustomValues.yaml file, enter:

  1	kubectl get persistentvolumeclaim

  The result shows the created volume:

  1 2	NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS data-myhivemq-0 Bound pvc-23b03202 300Gi RWO managed-premium

  
  For more information on Azure storage options for AKS, see

• Storage options for applications in Azure Kubernetes Service (AKS), Best practices for storage and backups in AKS, and Azure Pricing - Managed Disks.

Clean up

Your Azure account is billed for all services you create. Be sure to delete storage disks that are no longer in use. Deletion of your HiveMQ pods does not automatically delete the associated storage volumes.

• To view the persistent volume claims that are currently active, enter:

  1	kubectl get persistentvolumeclaim

• To delete a specific persistent volume claim and the associated persistent volume, enter the following command with the name of the PVC you want to delete. In our example, the name is data-myhivemq-0:

  1	kubectl delete persistentvolumeclaim data-myhivemq-0

Conclusion

The latest version of the HiveMQ Kubernetes Operator makes it easy to deploy a HiveMQ cluster using a StatefulSet template. When you use a StatefulSet, you need to configure the type of persistent storage the StatefulSet controller provisions for your HiveMQ data directory. For HiveMQ production use cases, your persistent storage volumes need to deliver a sustained performance of at least 900 IOPS. On AKS, you can specify Azure managed-premium SSD disks to ensure high IOPS for your MQTT workloads on HiveMQ. In general, the same tools and principles can be used with other Kubernetes as a Service providers (keep in mind that the specific storage class names and performance levels for each cloud vendor vary).

About Mary Brickenstein-Hofschen

Mary is a technical writer at HiveMQ. She enjoys providing precise information that helps HiveMQ users get things done.

Contact Mary