Introduction

HiveMQ offers a free and open source Extension SDK with service provider interfaces. The Extension System can be used to extend HiveMQ with custom business logic or to integrate virtually any system into HiveMQ.

Functionality that can be added with the help of the HiveMQ Extension System includes writing messages to a databases, integration of other service buses, collecting statistics, adding fine-grained security and virtually anything else you can imagine.

Extension development for HiveMQ is as easy as writing a Java main method once you grasp the core concepts. This documentation covers all relevant topics to get you started as quickly as possible.

HiveMQ extensions can interact with the MQTT broker in a variety of ways:

  • Use the Services API to interact with HiveMQ and the connected MQTT clients.

  • Register Callback Classes with HiveMQ that are called by HiveMQ when a certain Event occurs

  • Inspection and manipulation of MQTT related data, like sessions, retained messages, subscriptions and many more

  • Implement fine grained Authentication and Authorization for MQTT clients

  • Add a custom cluster discovery mechanism

  • Add custom logic to HiveMQ.

HiveMQ 3 Plugins are not compatible with HiveMQ 4.

For a list of already available extensions see the HiveMQ Marketplace.


Where to Start ?

The best way to start with extension development is to follow the Quickstart.

If you prefer to learn from real world extensions, visit the HiveMQ Github page for a variety of already existing extensions.

Advanced extension developers can delve right into the API JavaDoc or into the detailed chapters about the Concepts, Services and Registries to familiarize themselves with the SDK.


Quickstart

To get started with extension development, the first step is to become familiar with the basic structure of an HiveMQ extension.

Structure of an extension

An extension is a folder, which contains at least a Java Archive (.jar) and an XML descriptor file (hivemq-extension.xml).

Example Folder contents of an extension
|- hello-world-extension/  (1)
|   |- hello-world-extension-1.0.0.jar  (2)
|   |- hivemq-extension.xml (3)
1 The extensions folder. The folder name must match the id in the xml descriptor file
2 The Java Archive that contains the Java classes, which implement the extension’s logic
3 The XML descriptor file that contains additional information and metadata

The Java Archive contains all the compiled code of the extension, which implements the extension’s logic that is loaded and executed by HiveMQ. HiveMQ’s extension skeletons already come prepared with everything needed to automatically create this file from an extension’s source code.


Extension Information

The HiveMQ Extension SDK provides a dedicated XML file hivemq-extension.xml, which is used to store meta information as well as the extension’s priority.

The priority of an extension controls the order in which extensions are executed, if multiple extensions are installed. The highest priority is executed first.

We strongly recommend filling this file with all possible options, as HiveMQ will log this information, when the extension is loaded. This information can be useful for monitoring and debugging purposes.

Example hivemq-extension.xml content
<hivemq-extension>
    <id>hello-world-extension</id> (1)
    <name>Hello World Extension</name> (2)
    <version>1.0-SNAPSHOT</version> (3)
    <priority>1000</priority> (4)
    <author>dc-square-GmbH</author> (5)
</hivemq-extension>
1 The extension’s identifier. This identifier must be unique between all installed extensions.
2 The name for the extension. This name is displayed by HiveMQ.
3 The version of this extension.
4 The extension priority. (optional, default: 0)
5 The author of the extension. (optional)


Pre-requisites

The pre-requisites for starting the development of your personal custom HiveMQ Extension are minimal. All you need is a basic understanding of the Java programming language and the following two components.

  1. OpenJDK 11 or newer

  2. Maven

The use of a Java Development IDE like Eclipse or IntelliJ is strongly recommended.


Creating the Java project

There are multiple ways to create your first extension that you can choose from. For this guide the HiveMQ Maven Extension Archetype is used to create the basic skeleton for your extension.

To create the base for your extension, run the following command on the command line:

mvn archetype:generate -DarchetypeGroupId=com.hivemq -DarchetypeArtifactId=hivemq-extension-archetype -DarchetypeVersion=4.2.0

When prompted, specify the common Maven identifiers for the new project: GroupId, ArtifactId, Version, Package.

Example Output of the Maven Archetype Command
$ mvn archetype:generate -DarchetypeGroupId=com.hivemq -DarchetypeArtifactId=hivemq-extension-archetype -DarchetypeVersion=4.2.0

[INFO] Scanning for projects...
[INFO]
[INFO] ------------------< org.apache.maven:standalone-pom >-------------------
[INFO] Building Maven Stub Project (No POM) 1
[INFO] --------------------------------[ pom ]---------------------------------
[INFO]
[INFO] >>> maven-archetype-plugin:3.0.1:generate (default-cli) > generate-sources @ standalone-pom >>>
[INFO]
[INFO] <<< maven-archetype-plugin:3.0.1:generate (default-cli) < generate-sources @ standalone-pom <<<
[INFO]
[INFO]
[INFO] --- maven-archetype-plugin:3.0.1:generate (default-cli) @ standalone-pom ---
[INFO] Generating project in Interactive mode
[INFO] Archetype repository not defined. Using the one from [com.hivemq:hivemq-extension-archetype:4.2.0] found in catalog remote
Define value for property 'groupId': com.hivemq.extensions
Define value for property 'artifactId': hello-world-extension
Define value for property 'version' 1.0-SNAPSHOT: : 1.0.0
Define value for property 'package' com.hivemq.extensions: : com.hivemq.extensions.helloworld
Confirm properties configuration:
groupId: com.hivemq.extensions
artifactId: hello-world-extension
version: 1.0.0
package: com.hivemq.extensions.helloworld
 Y: : Y
[INFO] ----------------------------------------------------------------------------
[INFO] Using following parameters for creating project from Archetype: hivemq-extension-archetype:4.2.0
[INFO] ----------------------------------------------------------------------------
[INFO] Parameter: groupId, Value: com.hivemq.extensions
[INFO] Parameter: artifactId, Value: hello-world-extension
[INFO] Parameter: version, Value: 1.0.0
[INFO] Parameter: package, Value: com.hivemq.extensions.helloworld
[INFO] Parameter: packageInPathFormat, Value: com/hivemq/extensions/helloworld
[INFO] Parameter: package, Value: com.hivemq.extensions.helloworld
[INFO] Parameter: groupId, Value: com.hivemq.extensions
[INFO] Parameter: artifactId, Value: hello-world-extension
[INFO] Parameter: version, Value: 1.0.0
[INFO] Project created from Archetype in dir: /Users/cschaebe/Projects/play/docs/hello-world-extension
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 59.512 s
[INFO] Finished at: 2019-02-01T14:57:36+01:00
[INFO] ------------------------------------------------------------------------

A new folder named identical to the specified ArtifactId is created, which contains a fully functional HiveMQ extension. The Java Project for a HiveMQ extension is a basic Maven project that includes the hivemq-extension-sdk as dependency. The SDK is available on Maven Central and is automatically downloaded and added to your project by maven.

As an alternative to the Maven Archetype, you can also clone the hivemq-hello-world-extension from github.

Deploying an extension

At this point the extension is a fully functional HiveMQ extension project and the extension can be packaged and deployed to HiveMQ. Packaging an extension is as easy as executing the Maven goal package. Run the following command inside the extension folder to package the extension:

mvn package

This command creates a file <artifactId>-<version>-distribution.zip in the folder target. This zip archive contains a HiveMQ extension, which is ready to deploy.

To deploy your HiveMQ Extension to your HiveMQ instance, unzip the archive and move the extracted folder to the extensions folder of HiveMQ.

Install Extension

HiveMQ will automatically recognize and start the extension.

HiveMQ Console
2018-12-11 12:18:00,248 INFO  - Started Hello World Extension:1.0.0
2018-12-11 12:18:00,248 INFO  - Extension "Hello World Extension" version 1.0.0 started successfully.
HiveMQ 4 and its Extension SDK support Extension Hot Reload. This means you can add and remove Extensions to HiveMQ during runtime.

Developing an extension

Now that the extension can be packaged and deployed to HiveMQ, it is time to delve deeper into developing custom HiveMQ extensions.
Let’s start by exploring the essential parts of the Java project structure.


Java Project Structure

HiveMQ Extension Structure

Project Structure

A minimal HiveMQ extension consists of 1 Java class, a text file and an XML file.

  1. A class that implements com.hivemq.extension.sdk.api.ExtensionMain representing the starting point of every HiveMQ extension.

  2. A text file residing in META-INF/services named com.hivemq.extension.sdk.api.ExtensionMain that tells HiveMQ where to find this extension’s main class.

  3. An XML file in the resources folder named hivemq-extension.xml, which contains the extension’s metadata.

Using the Maven Archetype
The Maven Archetype automatically generates all these files with the desired content.


ExtensionMain

Starting point of every extension development is a class that implements the ExtensionMain class of the HiveMQ Extension SDK. The interface ExtensionMain consists of the methods extensionStart and extensionStop.

The method extensionStart is called by HiveMQ when the extension is started. This is also the starting point for extension development, since it is also the point where an extension can register its callbacks and implementations with HiveMQ.

Example

The following demonstrates the minimal contents of an HiveMQ Extensions Main class.

public class HelloWorldMain implements ExtensionMain {

    @Override
    public void extensionStart(ExtensionStartInput input, ExtensionStartOutput output) {
        //Code to run when extension is started
    }

    @Override
    public void extensionStop(ExtensionStopInput input, ExtensionStopOutput output) {
        //Code to run when extension is stopped
    }
}
When you rename the main class (here: HelloWorldMain), also remember to change the reference to the class in the file src/main/resources/META-INF/services/com.hivemq.extension.sdk.api.ExtensionMain.

Common Use-Cases

A HiveMQ extension virtually has infinite possibilities. This introduction focuses on some of the most common use-cases. At each point in the quickstart additional links are available, which provide a way to dive deeper into the details of the extension SDK and its APIs.

Some of the most common use-cases that can be implemented using the HiveMQ extension SDK are:


Authentication

Authentication is the mechanism of checking a communication partner’s identify. The most prominent mechanism of authentication is checking a persons ID card. In the context of MQTT, authentication is used to check if an MQTT client is allowed to connect to a broker and if it is allowed to use its chosen client identifier.

Implementing authentication in a HiveMQ extension is done by implementing a SimpleAuthenticator and registering an AuthenticatorProvider with HiveMQ. This is done in the extensionStart method of the main class. The custom provider for Authenticator is registered with the SecurityRegistry.

Registering Authenticator
public void extensionStart(ExtensionStartInput extensionStartInput, ExtensionStartOutput extensionStartOutput) {
    //register the provider with the SecurityRegistry
    Services.securityRegistry().setAuthenticatorProvider(new MyAuthenticatorProvider());
}

In this example the Authenticator Provider always returns a new Authenticator for every new MQTT Connection. It is also possible to share an instance of an Authenticator between multiple MQTT Connections, see this example.

Example Authenticator Provider
public class MyAuthenticatorProvider implements AuthenticatorProvider {

    @Override
    public Authenticator getAuthenticator(AuthenticatorProviderInput authenticatorProviderInput) {
        //return an instance of an Authenticator
        return new MyAuthenticator();
    }

}

A SimpleAuthenticator that implements username/password based authentication for MQTT clients uses the information from the input object to decide whether or not a client is allowed to connect.

In this example only clients with username "admin" and password "hivemq" are allowed to connect. Clients without username or password are not permitted.

Example Username/Password Authenticator
public class MyAuthenticator implements SimpleAuthenticator {

    @Override
    public void onConnect(SimpleAuthInput simpleAuthInput, SimpleAuthOutput simpleAuthOutput) {
        ...
        //check if the user is "admin" and the password is "hivemq"
        if (username.equals("admin") && password.equals("hivemq")) {
            simpleAuthOutput.authenticateSuccessfully();
        } else {
            simpleAuthOutput.failAuthentication();
        }
    }
}
Full Source of Example Username/Password Authenticator
public class MyAuthenticator implements SimpleAuthenticator {

    @Override
    public void onConnect(SimpleAuthInput simpleAuthInput, SimpleAuthOutput simpleAuthOutput) {

        //get the contents of the MQTT connect packet from the input object
        ConnectPacket connect = simpleAuthInput.getConnectPacket();

        //check if the client set username and password
        if (!connect.getUserName().isPresent() || !connect.getPassword().isPresent()) {
            simpleAuthOutput.failAuthentication();
        }

        //get username and password from the connect packet
        String username = connect.getUserName().get();
        String password = Charset.forName("UTF-8").decode(connect.getPassword().get()).toString();

        //check if the user is "admin" and the password is "hivemq"
        if (username.equals("admin") && password.equals("hivemq")) {
            simpleAuthOutput.authenticateSuccessfully();
        } else {
            simpleAuthOutput.failAuthentication();
        }
    }
}

In the Authenticator the extension developer has access to all information for a client that can be used for authentication. Including all the information from the client’s CONNECT packet and the client’s TLS certificate. This allows for even the most complex authentication mechanisms to be implemented.

Authorization

Authorization is a mechanism used to determine access control based on certain identities.
A human example for this are access cards, allowing entry to various areas of a building based on a person’s identity and permissions belonging to that identity. In an MQTT client context this 'access' refers to the ability of publishing and or subscribing to certain topics.

HiveMQ extensions support two mechanisms of authorization for MQTT publish and subscribe actions.

The simplest and most scalable mechanism are Default Permissions. The permissions for a client are represented by a list of topics and attributes that a client is allowed/denied to publish or subscribe to. Default permissions are independent for each client and can be either set in an Authenticator on in a ClientInitializer.

In this example each client is allowed to publish and subscribe only to topics that start with its own client identifier. Here we add a few lines to the existing Authenticator that add the client’s default permissions.

Default Permissions
...

//create a topic permission for all topics starting with the client identifier
final TopicPermission permission = Builders.topicPermission() (1)
        .topicFilter(simpleAuthInput.getConnectPacket().getClientId() + "/#") (2)
        .type(TopicPermission.PermissionType.ALLOW) (3)
        .build();

//add this permission to the client's default permissions
simpleAuthOutput.getDefaultPermissions().add(permission);

...
1 The extension SDK provides builders for classes that need to be constructed by an extension developer. The class Builders has static methods to access the specific builders.
2 Topic filter for this permission, the MQTT wildcards # and + can be used to create topic filters that match multiple topics.
3 A white-list approach is used here, to only allow access to specific topics. A black-list approach is also possible.
Full Authorizer with default permissions
public class MyAuthenticator implements SimpleAuthenticator {

    @Override
    public void onConnect(SimpleAuthInput simpleAuthInput, SimpleAuthOutput simpleAuthOutput) {

        //get the contents of the MQTT connect packet from the input object
        ConnectPacket connect = simpleAuthInput.getConnectPacket();

        //check if the client set username and password
        if (!connect.getUserName().isPresent() || !connect.getPassword().isPresent()) {
            simpleAuthOutput.failAuthentication();
        }

        //get username and password from the connect packet
        String username = connect.getUserName().get();
        String password = Charset.forName("UTF-8").decode(connect.getPassword().get()).toString();

        //check if the user is "admin" and the password is "hivemq"
        if (username.equals("admin") && password.equals("hivemq")) {

            //create a topic permission for all topics starting with the client identifier
            TopicPermission permission = Builders.topicPermission()
                    .topicFilter(simpleAuthInput.getConnectPacket().getClientId() + "/#")
                    .build();

            //add this permission to the client's default permissions
            simpleAuthOutput.getDefaultPermissions().add(permission);

            simpleAuthOutput.authenticateSuccessfully();
        } else {
            simpleAuthOutput.failAuthentication();
        }

    }

}
The permissions for an MQTT client can have more detail than only a topic (e.g. the QoS level or the retained flag) see the Topic Permission chapter for a list of all available permission options.

Default permissions provide access-control on a per topic basis. Another mechanism for authorization are Authorizers, these provide a way to implement authorization on a per-packet basis instead.

To learn more about Authorizers see the Publish Authorizer and Subscription Authorizer chapters.

Lifecycle Events

An extension can listen to Lifecycle Events to get notified when certain events in the lifecycle of an MQTT client occur. These events include a client establishing a connection, a client disconnecting, authentication of a client being successful, and more.

To create a lifecycle event listener, a ClientLifecycleEventListenerProvider is registered with the EventRegistry.

Register provider with EventRegistry
 Services.eventRegistry().setClientLifecycleEventListener(new MyClientLifecycleEventListenerProvider());

The Provider always returns a new instance of a ClientLifecycleEventListener for every client. It is also possible to share the same instance of the Listener for multiple clients, see this example.

Example ClientLifecycleEventListenerProvider
public class MyClientLifecycleEventListenerProvider implements ClientLifecycleEventListenerProvider {

    @Override
    public ClientLifecycleEventListener getClientLifecycleEventListener(ClientLifecycleEventListenerProviderInput input) {
        return new MyClientLifecyleEventListener();
    }
}

For this example a log statement is issued, when a client connects, authentication was successful and when a client disconnects.

Example ClientLifecycleEventListener
public class MyClientLifecyleEventListener implements ClientLifecycleEventListener {

    private static final Logger log = LoggerFactory.getLogger(MyClientLifecyleEventListener.class);

    @Override
    public void onMqttConnectionStart(ConnectionStartInput connectionStartInput) {
        log.info("Client {} connects.", connectionStartInput.getConnectPacket().getClientId());
    }

    @Override
    public void onAuthenticationSuccessful(AuthenticationSuccessfulInput authenticationSuccessfulInput) {
        log.info("Client {} authenticated successfully.", authenticationSuccessfulInput.getClientInformation().getClientId());
    }

    @Override
    public void onDisconnect(DisconnectEventInput disconnectEventInput) {
        log.info("Client {} disconnected.", disconnectEventInput.getClientInformation().getClientId());
    }
}
You can listen to more detailed events by overriding other methods from the interface ClientLifecycleEventListener.

See also


Intercept & Manipulate MQTT messages

A HiveMQ extensions can manipulate content of MQTT messages before they are processed by the broker through intercepting said messages. The developer can register a callback, called Interceptor, which is called by HiveMQ whenever a MQTT message is received.

In this example a PublishInterceptor is registered, that manipulates the topic of incoming publishes. An Interceptor is always registered in a ClientInitializer. The initializer is a callback that is called by HiveMQ when an MQTT client session is initialized.

Register ClientInitializer with the InitializerRegistry
Services.initializerRegistry().setClientInitializer(new MyClientInitializer());

An Interceptor is then added to the clientContext of the Initializer.

Register ClientInitializer with the InitializerRegistry
public class MyClientInitializer implements ClientInitializer {

    @Override
    public void initialize(InitializerInput initializerInput, ClientContext clientContext) {

        clientContext.addPublishInboundInterceptor(new PublishInboundInterceptor() { (1)
            @Override
            public void onInboundPublish(PublishInboundInput publishInboundInput, PublishInboundOutput publishInboundOutput) {
                //set the topic to "new/topic"
                publishInboundOutput.getPublishPacket().setTopic("new/topic"); (2)
            }
        });
    }
}
1 A new Interceptor is created and added to the client’s context.
2 The topic is changed here. Most of the attributes of a MQTT PUBLISH packet can be manipulated.
The Publish Interceptor can also be used to prevent onward delivery of incoming messages. See this example.

Publishing MQTT messages

It is not only possible to manipulate messages sent by clients but also to create and publish new messages from within an extension.

The Publish Service allows publishing messages to all subscribers for a topic or to a specific client (see this example).

Publish a MQTT message
Publish message = Builders.publish()
    .topic("the/topic")
    .qos(Qos.EXACTLY_ONCE)
    .payload(Charset.forName("UTF-8").encode("payload"))
    .build();

Services.publishService().publish(message);

A PUBLISH that is sent via the Publish Service is processed identical to a PUBLISH sent by a client.

See also


Modifying subscriptions

The Subscription Store allows extensions to manipulate a client’s subscriptions. There are methods to add/remove/get subscriptions for a client.

In this example a subscription for the topic new/topic with QoS 1 is added for the client with the identifier client-ID.

Adding a subscription for a client
TopicSubscription subscription = Builders.topicSubscription()
    .topicFilter("new/topic")
    .qos(Qos.AT_LEAST_ONCE)
    .build();

Services.subscriptionStore().addSubscription("client-ID", subscription);
When a subscription for a client is added via the extension system or HiveMQ Control Center, the retained message for the topics included in the subscription will not be published to the client.

Modifying Retained Messages

Retained messages can be set, removed and fetched in the extension SDK by utilizing the RetainedMessageStore.

For this example a retained message is added (or replaced) for the topic retain/topic with QoS 0 and a the payload content.

Adding a subscription for a client
RetainedPublish publish = Builders.retainedPublish()
    .topic("retain/topic")
    .payload(Charset.forName("UTF-8").encode("content"))
    .qos(Qos.AT_MOST_ONCE)
    .build();

Services.retainedMessageStore().addOrReplace(publish);
Retained messages that are added with the retained message store are not published to active subscribers, unlike retained messages sent by clients.

Cluster Discovery

HiveMQ’s cluster discovery can utilize custom discovery mechanisms provided by an extension. This allows a deep integration of a HiveMQ cluster into an existing infrastucture.

Register cluster discovery callback
Services.clusterService().addDiscoveryCallback(new MyClusterDiscovery());
Example cluster discovery
public class MyClusterDiscovery implements ClusterDiscoveryCallback {

    //this is a placeholder for an implementation of your custom external service
    MyExternalService externalService = new MyExternalService(); (1)

    @Override
    public void init(ClusterDiscoveryInput clusterDiscoveryInput, ClusterDiscoveryOutput clusterDiscoveryOutput) {
        externalService.registerNode(clusterDiscoveryInput.getOwnAddress()); (2)
        clusterDiscoveryOutput.provideCurrentNodes(externalService.getAllClusterNodes());
    }

    @Override
    public void reload(ClusterDiscoveryInput clusterDiscoveryInput, ClusterDiscoveryOutput clusterDiscoveryOutput) {
        clusterDiscoveryOutput.provideCurrentNodes(externalService.getAllClusterNodes()); (3)
    }

    @Override
    public void destroy(ClusterDiscoveryInput clusterDiscoveryInput) {
        externalService.removeNode(clusterDiscoveryInput.getOwnAddress()); (4)
    }
}
1 This class is just a placeholder for a custom external service
2 When this cluster node is started, it is registered with the external service
3 The reload method is called regularly and a list of addresses for all cluster nodes is provided to the output object. The reload interval can be configured.
4 When this cluster node is stopped, then this node is unregistered with the external service
If the cluster discovery from an extension shall be used, then HiveMQ’s cluster configuration must be set to extension. See this configuration example.


Metrics

HiveMQ extensions can access HiveMQ’s metrics and also provide their own metrics.

Metrics in HiveMQ extensions are handled by the widely used and well-known library Dropwizard Metrics.

Example read HiveMQ metric
Counter currentConnectionsCounter = Services.metricRegistry().counter("com.hivemq.networking.connections.current");
log.info("currently connected {}", currentConnectionsCounter.getCount());
Example add custom metric
final Counter customCounter = Services.metricRegistry().counter("demo.helloworld.started");
customCounter.inc();
There a multiple types of metrics available, including Gauges, Counters, Meters, Histogramms and Timers.

Next steps

  • Get acquainted with the APIs and Services that the extension SDK provides.

  • Look at existing extension code on github.

  • Implement your custom logic in a HiveMQ extension.