Introduction
HiveMQ offers a free, open-source extension SDK. The HiveMQ extension framework provides an open API that allows developers to create custom extensions that suit their specific infrastructures. Use our extension framework to extend HiveMQ with custom business logic or to integrate virtually any system into HiveMQ.
The HiveMQ extension framework can be used to add many types of functionality:
-
Write messages to a database,
-
integrate other services,
-
collect statistics,
-
add fine-grained security,
-
and much more.
Once you understand the core concepts, extension development for HiveMQ is as easy as writing a Java main method. This guide covers all the basics that you need to get started with extension development for HiveMQ.
HiveMQ extensions can interact with the MQTT broker in various ways:
-
Use our extension API to interact with HiveMQ and connected MQTT clients
-
Register callbacks with HiveMQ that HiveMQ calls when a specific event occurs
-
Inspect and manipulate MQTT-related data such as sessions, retained messages, subscriptions, and more
-
Implement fine-grained authentication and authorization for MQTT clients
-
Add a custom cluster-discovery mechanism
-
Add custom logic to HiveMQ
| HiveMQ 3 Plugins are not compatible with HiveMQ 4. |
For a list of all the HiveMQ extensions that are currenlty available, visit the HiveMQ Marketplace.
Get Started
The best way to begin extension development for HiveMQ is to follow our Quick Start Guide for HiveMQ Extension Developers.
If you prefer to learn from real-world extensions, visit the HiveMQ Github page to access a variety of current extensions.
For in-depth information about the HiveMQ Enterprise SDK, advanced extension developers can jump directly to the API JavaDoc or the detailed Concepts, Services, and Registries chapters of our documentation.
Quick Start Guide for HiveMQ Extension Developers
Extension Structure
An extension is a folder that contains at least a Java archive (.jar) and an XML descriptor file (hivemq-extension.xml).
|- hello-world-extension/ (1)
| |- hello-world-extension-1.0.0.jar (2)
| |- hivemq-extension.xml (3)| 1 | The name of the extension folder. This name must match the ID in your XML descriptor file |
| 2 | The Java archive that contains the Java classes that implement the logic of the extension |
| 3 | The XML descriptor file that contains additional information and metadata |
The Java archive contains all the compiled code of the extension. This code implements the logic that HiveMQ loads and executes. HiveMQ extension skeletons are preconfigured with everything that is needed to automatically create the XML-descriptor file from the source code of the extension.
Extension Information
The HiveMQ Extension SDK uses the hivemq-extension.xml to store meta information and the priority level of the extension.
The priority that you assign to an extension defines the order in which HiveMQ executes the extension. If multiple extensions are installed, the extension with the highest priority executes first.
| We strongly recommend that you fill in all of the metadata that is requested in the hivemq-extension.xml file. HiveMQ logs the information when the extension loads and the information is useful for monitoring and debugging purposes. |
<hivemq-extension>
<id>hello-world-extension</id> (1)
<name>Hello World Extension</name> (2)
<version>1.0-SNAPSHOT</version> (3)
<priority>1000</priority> (4)
<start-priority>1000</start-priority> (5)
<author>HiveMQ-GmbH</author> (6)
</hivemq-extension>| 1 | The extension identifier. This identifier must be unique across all of the HiveMQ extensions that are installed on a HiveMQ installation. |
| 2 | The name of the extension. HiveMQ displays this name for the extension throughout the HiveMQ installation. |
| 3 | The version of the extension. |
| 4 | The priority of the extension. The extension with the lowest priority executes last. The default setting is 0. This setting is optional. |
| 5 | The start priority of the extension. The extension with the lowest priority starts last. The default setting is 1000. The lowest possible priority is 0. This setting is optional. |
| 6 | The author of the extension. This setting is optional. |
| The start priority in the hivemq-extension.xml file is only used if it is not possible to determine the temporal resolution between two extension start events. This usually happens at the start of HiveMQ if several extensions are already present in the extensions folder, or during a simultaneous hot reload of multiple extensions. Otherwise the first extension that HiveMQ detects loads first. There are no guarantees for the order in which extensions stop. |
Prerequisites
All you need to start developing a custom HiveMQ extension is a basic understanding of the Java programming language and the following two components:
-
OpenJDK 11 or newer
| To facilitate your development process, we strongly recommend that you use a Java-development IDE such as Eclipse or IntelliJ. |
Create Your Java Project
There are many different ways to create your first extension for HiveMQ. In this guide, we use the HiveMQ Maven-extension archetype to create a base for the extension.
To begin, run the following command on the command line:
mvn archetype:generate -DarchetypeGroupId=com.hivemq -DarchetypeArtifactId=hivemq-extension-archetype -DarchetypeVersion=4.4.0When prompted, define the following Maven identifiers for your new project: GroupId, ArtifactId, Version, and Package.
$ mvn archetype:generate -DarchetypeGroupId=com.hivemq -DarchetypeArtifactId=hivemq-extension-archetype -DarchetypeVersion=4.4.0
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------< org.apache.maven:standalone-pom >-------------------
[INFO] Building Maven Stub Project (No POM) 1
[INFO] --------------------------------[ pom ]---------------------------------
[INFO]
[INFO] >>> maven-archetype-plugin:3.0.1:generate (default-cli) > generate-sources @ standalone-pom >>>
[INFO]
[INFO] <<< maven-archetype-plugin:3.0.1:generate (default-cli) < generate-sources @ standalone-pom <<<
[INFO]
[INFO]
[INFO] --- maven-archetype-plugin:3.0.1:generate (default-cli) @ standalone-pom ---
[INFO] Generating project in Interactive mode
[INFO] Archetype repository not defined. Using the one from [com.hivemq:hivemq-extension-archetype:4.4.0] found in catalog remote
Define value for property 'groupId': com.hivemq.extensions
Define value for property 'artifactId': hello-world-extension
Define value for property 'version' 1.0-SNAPSHOT: : 1.0.0
Define value for property 'package' com.hivemq.extensions: : com.hivemq.extensions.helloworld
Confirm properties configuration:
groupId: com.hivemq.extensions
artifactId: hello-world-extension
version: 1.0.0
package: com.hivemq.extensions.helloworld
Y: : Y
[INFO] ----------------------------------------------------------------------------
[INFO] Using following parameters for creating project from Archetype: hivemq-extension-archetype:4.4.0
[INFO] ----------------------------------------------------------------------------
[INFO] Parameter: groupId, Value: com.hivemq.extensions
[INFO] Parameter: artifactId, Value: hello-world-extension
[INFO] Parameter: version, Value: 1.0.0
[INFO] Parameter: package, Value: com.hivemq.extensions.helloworld
[INFO] Parameter: packageInPathFormat, Value: com/hivemq/extensions/helloworld
[INFO] Parameter: package, Value: com.hivemq.extensions.helloworld
[INFO] Parameter: groupId, Value: com.hivemq.extensions
[INFO] Parameter: artifactId, Value: hello-world-extension
[INFO] Parameter: version, Value: 1.0.0
[INFO] Project created from Archetype in dir: /Users/myuser/Projects/hello-world-extension
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 59.512 s
[INFO] Finished at: 2019-02-01T14:57:36+01:00
[INFO] ------------------------------------------------------------------------The Maven-archetype command creates a new folder that contains a fully functional HiveMQ extension.
This folder has the same name as the ArtifactId that you defined for the Maven-extension archetype of your project.
The Java project for a HiveMQ extension is a basic Maven project that includes the hivemq-extension-sdk as a dependency.
The SDK is available on Maven Central and is automatically downloaded and added to your project by Maven.
As an alternative to the Maven archetype, you can clone the hivemq-hello-world-extension on github.
|
Debug And Test Your Extension
When you develop a HiveMQ extension, regular testing is a fundamental part of the process. The HiveMQ Testcontainer gives you the tools to automatically deploy and run your extension on the fly inside a Docker container, while running a JUnit Test.
For more Information, see our guide on how to test HiveMQ extensions.
Deploy Your Extension
At this point in the process, the extension is a fully functional HiveMQ extension project that can be packaged and deployed to HiveMQ.
To package your extension, simply execute the following Maven goal in the extension folder:
mvn packageThis command creates an <artifactId>-<version>-distribution.zip archive in the target folder. The zip archive contains your ready-to-deploy HiveMQ extension.
To deploy your HiveMQ Extension to your HiveMQ instance, unzip the archive and move the extracted folder to the extensions folder of HiveMQ.
HiveMQ automatically recognizes and starts the extension.
2018-12-11 12:18:00,248 INFO - Started Hello World Extension:1.0.0
2018-12-11 12:18:00,248 INFO - Extension "Hello World Extension" version 1.0.0 started successfully.| HiveMQ 4 and the HiveMQ Extension SDK support Extension Hot Reload. This functionality allows you to add and remove extensions to HiveMQ during runtime. |
Develop Your Extension
Now that you can package and deploy your extension to HiveMQ, it’s time to delve deeper into the development of custom HiveMQ extensions.
Let’s start by exploring the essential parts of the Java project structure.
Java Project Structure
A minimal HiveMQ extension consists of a Java class, a text file, and an XML file:
-
A Java class that implements
com.hivemq.extension.sdk.api.ExtensionMainis the starting point of every HiveMQ extension. -
A text file in the META-INF/services folder named com.hivemq.extension.sdk.api.ExtensionMain tells HiveMQ where to find the main class of the extension.
-
An XML file in the resources folder named hivemq-extension.xml contains the metadata of the extension.
|
Using the Maven Archetype
The Maven Archetype automatically generates all of these files with the desired content.
|
ExtensionMain
The starting point of every extension development is a class that implements the ExtensionMain class of the
HiveMQ Extension SDK.
The interface ExtensionMain consists of the methods extensionStart and extensionStop.
HiveMQ calls the extensionStart method when you start the extension.
This method is also the starting point for extension development, since this is also the point where an extension can register callbacks and implementations with HiveMQ.
Example
The following example shows the minimal content of a main class for a HiveMQ extension:
public class HelloWorldMain implements ExtensionMain {
@Override
public void extensionStart(ExtensionStartInput input, ExtensionStartOutput output) {
//Code to run when extension is started
}
@Override
public void extensionStop(ExtensionStopInput input, ExtensionStopOutput output) {
//Code to run when extension is stopped
}
}
When you rename the main class (in the example, HelloWorldMain), remember to change the reference to the class in the src/main/resources/META-INF/services/com.hivemq.extension.sdk.api.ExtensionMain file.
|
Popular Use Cases
A HiveMQ extension has virtually infinite possibilities. This guide focuses on some of the most popular use cases. Throughout the guide, we provide links to help you dive deeper into the details of the extension SDK and associated APIs.
Here are some typical uses for the HiveMQ Extension SDK:
Authentication
Authentication is the mechanism that verifies the identity of your communication partner.
Air travel provides a classic example of authentication. Before you are permitted to board a plane, airport security checks your passport. Presentation of a passport that identifies you as the person whose name is on the ticket, allows you to proceed.
In the context of MQTT, authentication is used to check if an MQTT client is allowed to connect to a broker and if it is allowed to use a particular client identifier.
To apply authentication in a HiveMQ extension, you implement a SimpleAuthenticator and register an AuthenticatorProvider with HiveMQ.
This implementation is done in the extensionStart method of the main class. The custom provider for authenticator is registered with the SecurityRegistry.
public void extensionStart(ExtensionStartInput extensionStartInput, ExtensionStartOutput extensionStartOutput) {
//register the provider with the SecurityRegistry
Services.securityRegistry().setAuthenticatorProvider(new MyAuthenticatorProvider());
}In this example, the Authenticator Provider returns a new Authenticator for every new MQTT connection. It is also possible to share an instance of an authenticator between multiple MQTT connections, see this example.
public class MyAuthenticatorProvider implements AuthenticatorProvider {
@Override
public Authenticator getAuthenticator(AuthenticatorProviderInput authenticatorProviderInput) {
//return an instance of an Authenticator
return new MyAuthenticator();
}
}A SimpleAuthenticator that implements username/password based authentication for MQTT clients uses the information from the input object to decide whether or not a client is allowed to connect.
In this example, only clients with the username "admin" and password "hivemq" are allowed to connect. Clients that lack the username or password are not permitted.
public class MyAuthenticator implements SimpleAuthenticator {
@Override
public void onConnect(SimpleAuthInput simpleAuthInput, SimpleAuthOutput simpleAuthOutput) {
...
//check if the user is "admin" and the password is "hivemq"
if (username.equals("admin") && password.equals("hivemq")) {
simpleAuthOutput.authenticateSuccessfully();
} else {
simpleAuthOutput.failAuthentication();
}
}
}public class MyAuthenticator implements SimpleAuthenticator {
@Override
public void onConnect(SimpleAuthInput simpleAuthInput, SimpleAuthOutput simpleAuthOutput) {
//get the contents of the MQTT connect packet from the input object
ConnectPacket connect = simpleAuthInput.getConnectPacket();
//check if the client set username and password
if (!connect.getUserName().isPresent() || !connect.getPassword().isPresent()) {
simpleAuthOutput.failAuthentication();
}
//get username and password from the connect packet
String username = connect.getUserName().get();
String password = Charset.forName("UTF-8").decode(connect.getPassword().get()).toString();
//check if the user is "admin" and the password is "hivemq"
if (username.equals("admin") && password.equals("hivemq")) {
simpleAuthOutput.authenticateSuccessfully();
} else {
simpleAuthOutput.failAuthentication();
}
}
}In the authenticator, the extension developer has access to all client information that can be used for authentication. This information includes all of the information from the CONNECT packet and the TLS certificate of the client. Access to all available authentication information makes it possible to implement even the most complex authentication mechanisms.
Authorization
Authorization is the mechanism that controls access based on certain identities.
Let’s continue the air travel example that we began in the authentication section. After your ticket and passport authenticate your identity,
a boarding pass is used to authorizes your access to a specific plane, row, and seat.
In the context of an MQTT client, 'access' refers to the ability to publish and subscribe to certain topics.
HiveMQ extensions support two authorization mechanisms for the MQTT publish and subscribe actions:
-
Default Permissions: This mechanism provides access control on a per topic basis.
-
Authorizers: This mechanism implements authorization on a per-packet basis.
Default Permissions are the simplest and most scalable of the mechanisms. The permissions for a client are represented by a list of topics and attributes to which a client can or cannot publish or subscribe. Default permissions are independent for each client and can be set in an authenticator or in a ClientInitializer.
In this example, each client is allowed to publish and subscribe only to topics that start with the same client identifier as the client. Here, we add a few lines to the existing authenticator that add the default permissions of the client.
...
//create a topic permission for all topics starting with the client identifier
final TopicPermission permission = Builders.topicPermission() (1)
.topicFilter(simpleAuthInput.getConnectPacket().getClientId() + "/#") (2)
.type(TopicPermission.PermissionType.ALLOW) (3)
.build();
//add this permission to the client's default permissions
simpleAuthOutput.getDefaultPermissions().add(permission);
...| 1 | The extension SDK provides builders for classes that need to be constructed by an extension developer. The class Builders has static methods to access the specific builders. |
| 2 | Topic filter for this permission, the MQTT wildcards # and + can be used to create topic filters that match multiple topics. |
| 3 | A white-list approach that only allow access to specific topics is used in this example. A black-list approach is also possible. |
public class MyAuthenticator implements SimpleAuthenticator {
@Override
public void onConnect(SimpleAuthInput simpleAuthInput, SimpleAuthOutput simpleAuthOutput) {
//get the contents of the MQTT connect packet from the input object
ConnectPacket connect = simpleAuthInput.getConnectPacket();
//check if the client set username and password
if (!connect.getUserName().isPresent() || !connect.getPassword().isPresent()) {
simpleAuthOutput.failAuthentication();
}
//get username and password from the connect packet
String username = connect.getUserName().get();
String password = Charset.forName("UTF-8").decode(connect.getPassword().get()).toString();
//check if the user is "admin" and the password is "hivemq"
if (username.equals("admin") && password.equals("hivemq")) {
//create a topic permission for all topics starting with the client identifier
TopicPermission permission = Builders.topicPermission()
.topicFilter(simpleAuthInput.getConnectPacket().getClientId() + "/#")
.build();
//add this permission to the client's default permissions
simpleAuthOutput.getDefaultPermissions().add(permission);
simpleAuthOutput.authenticateSuccessfully();
} else {
simpleAuthOutput.failAuthentication();
}
}
}| The permissions for an MQTT client can include more detail than only the topic. For example, the QoS level or the retained-message flag. For a list of all available permission options, see the Topic Permission chapter. |
The other mechanism for authorization are Authorizers. Authorizers implement authorization on a per-packet basis.
To learn more about Authorizers, see the Publish Authorizer and Subscription Authorizer chapters.
Lifecycle Events
To get notified when certain events in the lifecycle of an MQTT client occur, an extension can listen to lifecycle events. These events include a client establishing a connection, a client disconnecting, successful authentication of a client, and more.
To create a lifecycle-event listener, register a ClientLifecycleEventListenerProvider with the EventRegistry.
Services.eventRegistry().setClientLifecycleEventListener(new MyClientLifecycleEventListenerProvider());The provider returns a new instance of a ClientLifecycleEventListener for each client. If desired, you can share the same instance of a listener between multiple clients. For more information, see this example.
public class MyClientLifecycleEventListenerProvider implements ClientLifecycleEventListenerProvider {
@Override
public ClientLifecycleEventListener getClientLifecycleEventListener(ClientLifecycleEventListenerProviderInput input) {
return new MyClientLifecyleEventListener();
}
}In this example, a log statement is issued when a client connects, when authentication is successful, and when a client disconnects.
public class MyClientLifecyleEventListener implements ClientLifecycleEventListener {
private static final Logger log = LoggerFactory.getLogger(MyClientLifecyleEventListener.class);
@Override
public void onMqttConnectionStart(ConnectionStartInput connectionStartInput) {
log.info("Client {} connects.", connectionStartInput.getConnectPacket().getClientId());
}
@Override
public void onAuthenticationSuccessful(AuthenticationSuccessfulInput authenticationSuccessfulInput) {
log.info("Client {} authenticated successfully.", authenticationSuccessfulInput.getClientInformation().getClientId());
}
@Override
public void onDisconnect(DisconnectEventInput disconnectEventInput) {
log.info("Client {} disconnected.", disconnectEventInput.getClientInformation().getClientId());
}
}
You can listen to more detailed events by overriding other methods from the interface ClientLifecycleEventListener.
|
Intercept & Manipulate MQTT Messages
To manipulate the content of an MQTT message before the broker processes the message, a HiveMQ extension can intercept the message.
The developer registers an Interceptor callback that HiveMQ calls each time an MQTT message is received.
In this example, a PublishInterceptor that manipulates the topic of incoming publishes is registered. An Interceptor is always registered in a ClientInitializer. The initializer is a callback that HiveMQ calls each time an MQTT client session is initialized.
Services.initializerRegistry().setClientInitializer(new MyClientInitializer());An Interceptor is then added to the clientContext of the Initializer.
public class MyClientInitializer implements ClientInitializer {
@Override
public void initialize(InitializerInput initializerInput, ClientContext clientContext) {
clientContext.addPublishInboundInterceptor(new PublishInboundInterceptor() { (1)
@Override
public void onInboundPublish(PublishInboundInput publishInboundInput, PublishInboundOutput publishInboundOutput) {
//set the topic to "new/topic"
publishInboundOutput.getPublishPacket().setTopic("new/topic"); (2)
}
});
}
}| 1 | A new Interceptor is created and added to the context of the client. |
| 2 | The changed topic. Most of the attributes of an MQTT PUBLISH packet can be manipulated. |
| The Publish Interceptor can also be used to prevent onward delivery of incoming messages. For more information, see this example. |
Publish MQTT Messages
You can use an extension to manipulate the messages clients send or to create and publish new messages.
The Publish Service allows you to publish messages to all subscribers of a topic or to a specific client. For more information, see this example.
Publish message = Builders.publish()
.topic("the/topic")
.qos(Qos.EXACTLY_ONCE)
.payload(Charset.forName("UTF-8").encode("payload"))
.build();
Services.publishService().publish(message);The PUBLISH messages that the Publish Service sends are processed in the same way as a PUBLISH message that a client sends.
Modify Subscriptions
The Subscription Store allows extensions to manipulate the subscriptions of a client. The Subscription Store contains methods to add, remove, or get subscriptions for a client.
In this example, a subscription for the topic new/topic with QoS 1 is added for the client with the identifier client-ID.
TopicSubscription subscription = Builders.topicSubscription()
.topicFilter("new/topic")
.qos(Qos.AT_LEAST_ONCE)
.build();
Services.subscriptionStore().addSubscription("client-ID", subscription);| When a subscription for a client is added via the extension system or the HiveMQ Control Center, the retained message for the topics that are included in the subscription are not published to the client. |
Modify Retained Messages
In the extension SDK, the RetainedMessageStore can set, remove, and fetch retained messages.
For this example, a retained message is added (or replaced) for the topic retain/topic with QoS 0 and the payload content.
RetainedPublish publish = Builders.retainedPublish()
.topic("retain/topic")
.payload(Charset.forName("UTF-8").encode("content"))
.qos(Qos.AT_MOST_ONCE)
.build();
Services.retainedMessageStore().addOrReplace(publish);| Unlike the retained messages that clients send, retained messages that are added with the Retained Message Store are not published to active subscribers. |
Cluster Discovery
The HiveMQ cluster discovery feature can utilize the custom discovery mechanisms that an extension provides. This functionality enables deep integration of a HiveMQ cluster into an existing infrastructure.
Services.clusterService().addDiscoveryCallback(new MyClusterDiscovery());public class MyClusterDiscovery implements ClusterDiscoveryCallback {
//this is a placeholder for an implementation of your custom external service
MyExternalService externalService = new MyExternalService(); (1)
@Override
public void init(ClusterDiscoveryInput clusterDiscoveryInput, ClusterDiscoveryOutput clusterDiscoveryOutput) {
externalService.registerNode(clusterDiscoveryInput.getOwnAddress()); (2)
clusterDiscoveryOutput.provideCurrentNodes(externalService.getAllClusterNodes());
}
@Override
public void reload(ClusterDiscoveryInput clusterDiscoveryInput, ClusterDiscoveryOutput clusterDiscoveryOutput) {
clusterDiscoveryOutput.provideCurrentNodes(externalService.getAllClusterNodes()); (3)
}
@Override
public void destroy(ClusterDiscoveryInput clusterDiscoveryInput) {
externalService.removeNode(clusterDiscoveryInput.getOwnAddress()); (4)
}
}| 1 | This class is a placeholder for a custom external service. |
| 2 | When this cluster node starts, the node registers with the external service. |
| 3 | The reload method is called regularly and a list of addresses for all cluster nodes is provided to the output object. You can configure the reload interval. |
| 4 | When this cluster node stops, the node is unregistered with the external service. |
If you want to use cluster discovery from an extension, the cluster configuration of HiveMQ must be set to extension.
For more information, see configuration example.
|
Metrics
HiveMQ extensions can provide their own metrics and access HiveMQ metrics.
Metrics in HiveMQ extensions are handled by the widely-used and well-known Dropwizard Metrics library.
Counter currentConnectionsCounter = Services.metricRegistry().counter("com.hivemq.networking.connections.current");
log.info("currently connected {}", currentConnectionsCounter.getCount());final Counter customCounter = Services.metricRegistry().counter("demo.helloworld.started");
customCounter.inc();| Many types of metrics are available. Metric types include: Gauges, Counters, Meters, Histograms, and Timers. |