HiveMQ on DC/OS
Written by Simon Baier
Published: September 10, 2019
In today’s world we as customers have been conditioned to expect flawless availability with outstanding user experience from all the great companies delivering digital services catering to our every needs, such as convenience, entertainment, and security. These services and the quality of delivery customers expect now are only possible via the use of sophisticated, distributed systems and seamlessly connecting many services together, which puts an enormous burden upon system architects and especially operators. Technologies like DC/OS, a distributed operating system based on the Apache Mesos distributed systems kernel, are therefore becoming increasingly popular and important. DC/OS turns multiple machines into a single logical computer, significantly simplifying the management of the system, as well as the management and installation of distributed services across the system.
For user convenience DC/OS provides a great ecosystem in their service catalog, from where you can easily spawn a huge variety of services. Typically these services are either developed by D2IQ directly or by the developers of the application that will be run as a service. It is good practise to include most or all operational knowledge required within these DC/OS services. This pattern is also used in other orchestration and automation technologies like Kubernetes’ Operator pattern.
With this in mind the HiveMQ team has decided to develop our own, official HiveMQ service for DC/OS. This means if you are running DC/OS, adding a HiveMQ service is now as easy as running a single command, or clicking “Install” while browsing the service catalog. The purpose of this post is to illustrate how to deploy HiveMQ on DC/OS and putting it into context with existing services.
Installing HiveMQ on DC/OS
To install HiveMQ on DC/OS, navigate to the Catalog on your DC/OS dashboard and type HiveMQ. Click “Review & Run” and configure the service as needed. Confirm with “Run Service” and DC/OS will take care of launching the HiveMQ nodes.
Hint: You can also install and customize the service using the DC/OS CLI, see the DC/OS example
Note: For the following chapters, keep in mind that <service-name> is the name of your installation of the service. The default value for this parameter is hivemq.
Connecting Clients to HiveMQ on DC/OS
The Mesosphere DC/OS HiveMQ service provides several endpoints for connecting MQTT clients. The easiest way to connect clients, if you are running DC/OS Enterprise, is using Edge-LB.
In case you want to roll out your own load balancing solution or are not running DC/OS Enterprise, there are other ways to connect to your HiveMQ cluster from the edge.
For example, you can use the Virtual IP address provided by the service.
Another possibility is utilizing the SRV Records provided by Mesos-DNS.
After setting up Edge-LB, create a pool using the following hivemq-pool.json:
This pool will spawn a single load balancer on each of your public DC/OS nodes. Clients can then connect to the one of the load balancers, which will distribute the load across all HiveMQ of your cluster. This example will use the default mqtt port. You can also configure the pool to forward connections to the WebSocket- or TLS listeners, if they are enabled.
Hint: By default, Edge-LB will allow 10,000 concurrent connections per instance. To change this, you will need to use the Edge-LB template commands to dump and update the maxconn parameters in the template.
Hint: To allow more than 50,000 connections, you should run multiple instances (increase the count of the pool):
Hint: This type of TLS support is only available on DC/OS Enterprise.
Caution: If you want to use TLS on your cluster transport, you must deploy the service with cluster TLS enabled. The service will prohibit changes to this configuration after initial deployment to avoid data loss.
To enable TLS, you must first create a service account using your private and public keys:
Configure your service to use the service account <service-name>-principal and secret <service-name>/account-secret. Use the DC/OS Dashboard to update your configuration or use an options.json. For example:
See Authenticating DC/OS Services for more information.
Integrating HiveMQ in your DC/OS Monitoring
Monitoring is a crucial part for operations of any kind of service.
For Typical IoT use case with ten of thousands of clients and intense message throughput this is especially true.
DC/OS provides a quick and uncomplicated way of using Prometheus and Grafana to monitor your HiveMQ cluster.
First follow the steps at Export DC/OS Metrics to Prometheus to set up Prometheus and Grafana on DC/OS.
After setting up your monitoring deployment, log into your Grafana instance and add the Prometheus data source.
Finally you can import the HiveMQ Dashboard to Grafana.
Contact Us for recommendations and guidance towards the best monitoring approach for your specific use case.
Custom HiveMQ extensions
As mentioned in the opening paragraph of this post, modern service architecture in almost all cases consists of numerous important systems that need to be integrated with one another. DC/OS delivers improved operational and managerial integration of multiple services. HiveMQ with its Java based Extension SDK provides an open source way of integrating services on application level. You can install, delete and manage the configuration of any HiveMQ Extension, with the HiveMQ DC/OS service.
Note: To ensure consistency on all nodes in your cluster make sure to re-apply plans to any node you may add to the service for scaling in the future.
Adding HiveMQ extensions
To install an extension, you can use the add-extension plan. This plan requires a single parameter URL which requires a path to a ZIP compressed extension folder.
For example, to manually install the file RBAC extension on each current cluster node, run:
Deleting HiveMQ extensions
You can also delete extensions using the delete-extension plan. This plan requires the sole parameter EXTENSION, which corresponds to the extension’s folder name.
Adding or updating HiveMQ extension configuration
To configure an extension, you can update or add configuration files using the add-config
For example, to manually configure the RBAC extension file credentials.xml on each currently active cluster node, run:
Enable / disable HiveMQ extensions
Extensions can also be enabled or disabled on any cluster node during runtime as well. To do so, you can use the enable-extension or disable-extension plans. Both plans require the parameter EXTENSION parameter which corresponds to the extension’s folder name, e.g.
These HiveMQ extension plans, enable you to integrate your HiveMQ Enterprise cluster with other services such as Kafka by installing the Kafka extension
Installing new licenses
Note: If you plan to update after you deploy an updated license in this way, make sure to define the new license as the default before you perform the update.
Similar to extension management, you can also add new licenses to an already deployed cluster. You can use the add-license plan do to so. This plan requires the parameters LICENSE and LICENSE_NAME to be defined, where LICENSE is your base64 encoded license file and LICENSE_NAME is the name of the license file that is created.
For additional, in-depth information regarding the DC/OS HiveMQ service, check out the HiveMQ DC/OS service on GitHub
With the HiveMQ DC/OS service, we provide a native solution to introducing HiveMQ as the central messaging broker for any DC/OS based system or platform. With the use of HiveMQ Extensions like the commercial HiveMQ Enterprise Security Extension or the HiveMQ Kafka Extension, which can be installed and managed through DC/OS, you can seamlessly integrate the MQTT broker cluster with other applications and services. Contact Us, if you have any additional questions or are looking for guidance on integrating HiveMQ with other container or orchestration technologies.
Have a great day,
Simon from the HiveMQ Team