Skip to content

Implementing MQTT Challenge-Response Authentication

by Yannick Weber
7 min read

Since HiveMQ 4.3, you can define custom enhanced authentication flows in your HiveMQ extension.

MQTT 5 enhanced authentication gives you the tools you need to implement challenge-response style authentication. In contrast to traditional credential-based approaches, the server authenticates a client by presenting a challenge that the client must respond to with a valid response.

Although the implementation is more complicated, challenge-response authentication has several advantages. For instance, the ability to implement client authentication without the direct exchange of authentication secrets.

The goal of this demonstration is to only authenticate clients that can solve basic math problems. Upon connection, we present the client with a math equation that the client must solve. If we receive a response that contains the correct solution, we authenticate the client.

Authentication Flow

To initiate our enhanced authentication flow, the client sends a CONNECT message that contains mathChallenge as the authentication method. HiveMQ then decides to continue the authentication and sends the challenge in an AUTH message. The client retrieves the challenge, tries to complete the math equation and sends a response to HiveMQ. HiveMQ checks if the result matches the expected response and decides whether or not to authenticate the client. A successful authentication flow looks something like this:

Challenge-Response Authentication FlowChallenge-Response Authentication Flow

Implementing the Extension

To implement the extension, we refer to the HiveMQ Extension Developer Guide to show us how to create an empty extension where we can put our authentication logic.

Implementing the Authenticator

The authenticator component is responsible for handling the authentication requests and deciding whether or not a user is authenticated. For our purpose, we need to implement the methods onConnect and onAuth. The onConnect method gets called every time a client connects. In this method, we do the following:

  • Retrieve the authentication method from the CONNECT message (1)

  • Check if the authentication method is present and equal to mathChallenge otherwise fail the authentication (2)

  • Compute the parameters of our challenge that we present to the client (3)

  • Store the result of the challenge in the ConnectionAttributeStore, so we can retrieve it later on handling the response in the onAuth method (4)

  • Send the challenge to the client (5)

public class MathChallengeAuthenticator implements EnhancedAuthenticator {


    public void onConnect(
            final @NotNull EnhancedAuthConnectInput input,
            final @NotNull EnhancedAuthOutput output) {

        final Optional<String> authenticationMethod =
                input.getConnectPacket().getAuthenticationMethod(); (1)
        if (authenticationMethod.isPresent() 
            && "mathChallenge".equals(authenticationMethod.get())) { (2)
            final int first = random.nextInt();
            final int second = random.nextInt();
            final String challenge = first + "+" + second;
            final String expected = "" + (first + second); (3)

            final ConnectionAttributeStore store =
            store.putAsString("mathChallengeExpected", expected); (4)

            output.continueAuthentication(challenge.getBytes(StandardCharsets.UTF_8)); (5)


The onAuth method gets called when an AUTH message is received. In our example, we handle AUTH packages in the following way:

  • Retrieve the authentication method from the AUTH message (1)

  • Check if the authentication method is equal to mathChallenge otherwise fail the authentication (2)

  • Retrieve the expected response from the ConnectionAttributeStore (3)

  • Retrieve the client response from the authentication data in the AUTH message (4)

  • Verify that the expected response and the actual response are present, otherwise fail the authentication (5)

  • If the expected response and the actual response are equal, the authentication is successful (6)

public class MathChallengeAuthenticator implements EnhancedAuthenticator {


    public void onAuth(
            final @NotNull EnhancedAuthInput input,
            final @NotNull EnhancedAuthOutput output) {

        final String authenticationMethod =
            input.getAuthPacket().getAuthenticationMethod(); (1)

        if ("mathChallenge".equals(authenticationMethod)) { (2)

            final ConnectionAttributeStore store =

            final Optional<String> mathChallengeExpected =
                store.getAsString("mathChallengeExpected"); (3)
            final Optional<byte[]> authenticationData =
                input.getAuthPacket().getAuthenticationDataAsArray(); (4)

            if (mathChallengeExpected.isEmpty() || authenticationData.isEmpty()) { (5)

            if (mathChallengeExpected.get().equals(new String(authenticationData.get()))) { (6)



TIP: It is a good practice to use failAuthentication as the last fallback at the end of the authentication method. But make sure that you only call one of these decisive methods (fail, continue, etc.).

Registering the Authenticator

After we implement our custom authenticator, we need it to be recognized by HiveMQ. To achieve this, we simply register a provider with the SecurityRegistry, that returns our authenticator.

public void ExtensionStart(
        final @NotNull ExtensionStartInput input,
        final @NotNull ExtensionStartOutput output) {

    //register the provider with the SecurityRegistry
        .setEnhancedAuthenticatorProvider(i -> new MathChallengeAuthenticator());

Implementing the Client Application

Now that we have a running HiveMQ extension that supports the math challenge authentication, we want a client to actually test our extension. For that, we use the HiveMQ MQTT client. First, we need to implement an EnhancedAuthenticationMechanism that defines the following:

  • The authentication method that is sent with the CONNECT message (1)

  • The timeout of the authentication (2)

When the server continues the authentication, we handle the incoming AUTH message in the ‘onContinue’ method :

  • Retrieve the challenge, contained in the authentication data (3)

  • Check if the authentication data is present and the authentication method is correct (4)

  • Compute the challenge (5)

  • Add the response to the outgoing AUTH message (6)

  • Complete successfully (7)

private class MathChallengeEnhancedAuthMechanism implements Mqtt5EnhancedAuthMechanism {


    public @NotNull MqttUtf8String getMethod() {
        return MqttUtf8String.of("mathChallenge"); (1)

    public int getTimeout() {
        return (int) Duration.ofMinutes(3).getSeconds(); (2)

    public @NotNull CompletableFuture<Boolean> onContinue(
            final @NotNull Mqtt5ClientConfig clientConfig,
            final @NotNull Mqtt5Auth auth,
            final @NotNull Mqtt5AuthBuilder authBuilder) {

        final Optional<ByteBuffer> authData = auth.getData(); (3)

        if (authData.isPresent() && "mathChallenge".equals(auth.getMethod().toString())) { (4)

            final byte[] array = new byte[authData.get().remaining()];

            final String challenge = new String(array);

            final String[] split = challenge.split("\\+");

            final String response =
                "" + (Integer.parseInt(split[0]) + Integer.parseInt(split[1])); (5)

  ; (6)

            return CompletableFuture.completedFuture(true); (7)
        return CompletableFuture.completedFuture(false);

    public @NotNull CompletableFuture<Boolean> onAuthSuccess(
            final @NotNull Mqtt5ClientConfig clientConfig,
            final @NotNull Mqtt5ConnAck connAck) {

        return CompletableFuture.completedFuture(true);


At this point, we only need to register the enhanced authentication method before we connect our client:

public class Client {

    public static void main(final @NotNull String[] args) {
        final Mqtt5Client client = Mqtt5Client.builder()
                .enhancedAuth(new MathChallengeEnhancedAuthMechanism())




Although the math challenge authentication is very good for demonstration purposes, it is not very well suited for real-life use cases. However, the HiveMQ 4.3 Extension SDK provides a good abstraction of MQTT 5 Enhanced Authentication which makes it very easy to implement a complex authentication procedure yourself.

You can find the code for the extension and the client on GitHub.

Yannick Weber

Yannick is a Senior Software Engineer and one of the core members of HiveMQ's product development team. He has a strong interest in messaging technologies and is focusing on quality development of HiveMQ's many tools and extensions. In addition, he is the maintainer of the HiveMQ module in the testcontainers-java project.

  • Contact Yannick Weber via e-mail
HiveMQ logo
Review HiveMQ on G2