HiveMQ Enterprise Security Extension

The HiveMQ Enterprise Security Extension (ESE) expands the role, user, and permission-management capabilities of HiveMQ Enterprise and Professional editions. ESE allows you to use different sources of external authentication and authorization data to authenticate and authorize MQTT clients. In the ESE, you define realms to partition your server into protected areas that can each have their own authentication and / or authorization scheme. The ESE processes incoming client connections in highly configurable pipelines that offer customizable stages to handle the authentication and authorization of your clients.

If you are unfamiliar with MQTT and HiveMQ security concepts, we highly recommend this article: https://www.hivemq.com/mqtt-security-fundamentals/

Features

Requirements

  • A running HiveMQ Professional or Enterprise Edition installation (versions 4.1.0 and higher)

  • A running SQL database

Installation

  1. Download the HiveMQ Enterprise Security Extension from our marketplace.

  2. Unzip the downloaded archive and go to the hivemq-enterprise-security-extension folder.

  3. Configure the ESE in the enterprise-security-extension.xml file that is contained in the ESE folder.

  4. Move the ESE folder into the extensions folder of your HiveMQ installation.

└─ <HiveMQ folder>
    ├─ bin
    ├─ config
    ├─ data
    ├─ extensions
    │   ├─ hivemq-enterprise-security-extension
    │   └─ ...
    ├─ license
    ├─ log
    └─ ...
To secure an entire HiveMQ cluster, the HiveMQ Enterprise Security Extension must be installed on every HiveMQ broker node in the cluster.

To verify that the extension is installed and configured properly, check your HiveMQ logs.

If you want to change the configuration of the ESE, you can add a DISABLED file to the ESE folder to disable the extension. To enable ESE once more, remove the DISABLED file that you added.


Configuration

The Enterprise Security Extension needs to be configured to use your particular authentication and authorization sources and meet your individual requirements.

Configuration is done in the enterprise-security-extension.xml file that is located in the ESE extension folder. To facilitate configuration and validation, the ESE provides an enterprise-security-extension.xsd schema file next to the configuration file.

Configuration file example
<?xml version="1.0" encoding="UTF-8" ?>
<enterprise-security-extension
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="enterprise-security-extension.xsd"
        version="1">
    <realms>
        <!-- a postgresql db-->
        <sql-realm>
            <name>postgres-backend</name>
            <enabled>true</enabled>
            <configuration>
                <db-type>POSTGRES</db-type>
                <db-name>hivemq</db-name>
                <db-host>hostname</db-host>
                <db-port>5432</db-port>
                <db-username>hivemq</db-username>
                <db-password>password</db-password>
            </configuration>
        </sql-realm>
    </realms>
    <pipelines>
        <!-- secure access to the mqtt broker -->
        <listener-pipeline listener="ALL">
            <!-- authenticate over a sql db -->
            <sql-authentication-manager>
                <realm>postgres-backend</realm>
            </sql-authentication-manager>
            <!-- authorize over a sql db -->
            <sql-authorization-manager>
                <realm>postgres-backend</realm>
                <use-authorization-key>false</use-authorization-key>
                <use-authorization-role-key>true</use-authorization-role-key>
            </sql-authorization-manager>
        </listener-pipeline>
        <!-- secure access to the control center -->
        <control-center-pipeline>
            <!-- authenticate over a sql db -->
            <sql-authentication-manager>
                <realm>postgres-backend</realm>
            </sql-authentication-manager>
            <!-- authorize over a sql db -->
            <sql-authorization-manager>
                <realm>postgres-backend</realm>
                <use-authorization-key>true</use-authorization-key>
                <use-authorization-role-key>true</use-authorization-role-key>
            </sql-authorization-manager>
        </control-center-pipeline>
    </pipelines>
</enterprise-security-extension>

The root tag in the configuration file is called <enterprise-security-extension>. This tag encloses the configurations for your access log, realms, and pipelines. For more information, refer to the individual sections.

The following snippet shows the overall structure of the configuration file:

Configuration file structure
<enterprise-security-extension
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="enterprise-security-extension.xsd"
        version="1">

    <access-log>
        <!-- optional access log configuration -->
    </access-log>
    <realms>
        <!-- add one or more realm configurations -->
    </realms>
    <pipelines>
        <!-- add one or more pipeline configurations -->
    </pipelines>
</enterprise-security-extension>


Concepts

Realm

Realms are abstract representations of external sources of authentication or authorization information. The ESE uses realms to authenticate connecting clients and authorize the future actions of the connecting clients. Specific types of realms correspond to the different types of authentication and authorization sources that are available. For example, an SQL realm uses an SQL database as its source.

Each type of realm has a specific configuration.

Realm configuration structure
<enterprise-security-extension>
    ...

    <realms>
        <sql-realm> (1)
            <name>realm-name</name>
            <enabled>true</enabled>
            <configuration>
                <!-- enter specific configuration options, for example, database url and credentials -->
            </configuration>
        </sql-realm>

        <!-- configure additional realms -->
    </realms>

    ...
</enterprise-security-extension>
1 sql-realm is used as an example here

In general, realms need to establish connections to external services. You must ensure that these external services are available for connection establishment when you start the extension.

Realms are configured in the <realms> tag of the ESE configuration file. Each realm needs a unique name so it can be accurately referenced in your pipelines. Since establishing a realm connection can be an expensive operation, all realms can be disabled for testing purposes.

Available Realm Types

Table 1. Overview over the Available Realm Types

Realm

MQTT Client

Control-Center

Authentication

Authorization

Authentication

Authorization

SQL Realm


Pipeline

When a client tries to connect to HiveMQ, the client runs through a pipeline of the Enterprise Security Extension before the connection is accepted. This pipeline contains the individual steps that are required to authenticate and authorize the client. It is possible to configure multiple pipelines to authenticate or authorize different types of clients with different mechanisms.

Pipelines are highly configurable. However, a pipeline usually has four stages:

  • Authentication preprocessing

  • Authentication

  • Authorization preprocessing

  • Authorization

MQTT client CONNECT sequence

Authentication Preprocessing

Preprocessing is done with ESE variables. When a client connects at the start of a pipeline, only the MQTT variables (mqtt-clientid, mqtt-username, mqtt-password) are filled with the values the client provides in its CONNECT packet. The purpose of authentication preprocessing is to fill the authentication variables that the next stage of the pipeline requires to actually authenticate the client. The way variables are preprocessed and transformed is explained in the Preprocessing section.

Authentication preprocessing is optional. By default, the MQTT user name and password are filled in as the authentication key and authentication secret.

Authentication

Authentication is performed by an authentication manager. For example, an SQL authentication manager that uses a database realm. The authentication manager uses the authentication variables to verify whether a client is authenticated to connect to HiveMQ. Additionally, the authentication manager can provide values for some authorization variables. For example, provide a role key that can be used for authorization.

Authorization Preprocessing

Preprocessing for authorization is similar to authentication preprocessing. The only difference is that authorization preprocessing fills or transforms the authorization variables.

Authorization

Authorization is performed by an authorization manager. For example, an SQL authorization manager that uses a database realm. The authorization manager uses authorization variables to query permissions for a client. These permissions define which operations the client is allowed to perform.

Pipeline configuration structure
<enterprise-security-extension>
    ...

    <pipelines>
        <listener-pipeline listener="ALL">
            <authentication-preprocessors>
                ...
            </authentication-preprocessors>

            <sql-authentication-manager>
                ...
            </sql-authentication-manager>

            <authorization-preprocessors>
                ...
            </authorization-preprocessors>

            <sql-authorization-manager>
                ...
            </sql-authorization-manager>
        </listener-pipeline>

        <!-- configurations for additional pipelines -->
    </pipelines>

</enterprise-security-extension>


Listener Pipeline

When a client connects to HiveMQ, the Enterprise Security Extension selects the appropriate pipeline for the client.

It is possible to configure multiple pipelines that each map to different HiveMQ listeners. This enables you to authenticate and authorize backend applications and devices that connect over the internet differently.

Listener pipelines configuration
<enterprise-security-extension>
    ...

    <pipelines>
        <listener-pipeline listener="listener-internet"> (1)
            ...
        </listener-pipeline>

        <listener-pipeline listener="listener-backend">
            ...
        </listener-pipeline>

        <listener-pipeline listener="ALL"> (2)
            ...
        </listener-pipeline>
    </pipelines>

</enterprise-security-extension>
1 The listener attribute takes the name of the listener that is defined in the HiveMQ config.xml.
2 ALL is a keyword for a special pipeline. This pipeline is used if a client connects to a listener whose name does not match another pipeline.

If a client connects to a listener whose name does not match any pipeline and no ALL pipeline is configured, the ESE rejects the connection attempt.

Pipeline selection process


ESE Variables

The Enterprise Security Extension assigns a set of ESE variables to every connecting client. To give you maximum authentication and authorization flexibility, you can configure how the ESE variables are manipulated in the preprocessing stages of a pipeline.

There are two data types available for ESE variables:

  • String: ESE variables of this data type store a sequence of characters of arbitrary length or null

  • Byte: ESE variables of this data type store a sequence of bytes of arbitrary length or null

Additionally, ESE variables can be constant. Constant ESE variables contain read-only values that the ESE runtime system provides. Specific preprocessors can only use specific types of ESE variables.

Some ESE variables have predefined meanings, others can be used as general purpose variables. ESE variables are grouped into four categories:

MQTT Variables

MQTT variables are initialized by the ESE runtime system. The value of an MQTT variable is derived directly from the MQTT CONNECT packet of the connecting client.

Table 2. MQTT variables
Name Data type Description

mqtt-clientid

Constant string

Set to the client ID of the connecting client.

mqtt-username

Constant string

Set to the user name field of the connecting client.

mqtt-password

Constant byte

Set to the password field of the connecting client.

General Purpose Variables

General purpose variables do not have a predefined meaning. These variables can be used during pipeline processing as desired. At the start of a pipeline, the initial state of a general purpose variable is always null.

Table 3. General purpose variables
Name Data type

byte-1 byte-2 byte-3 byte-4 byte-5 byte-6

Byte

string-1 string-2 string-3 string-4 string-5 string-6

String

Authentication Variables

Authentication variables are used by the authentication managers to verify the identity of a connecting client. These variables are usually manipulated in the authentication-preprocessing step of a pipeline.

Table 4. Authentication variables
Name Data type Description

authentication-key

String

The authentication-key is generally used by the authentication manager to identify the connecting client.

authentication-byte-secret

Byte

The authentication-byte-secret is generally used by the authentication manager to ensure the claims of the connecting client.

Authorization Variables

Authorization variables are used by the authorization managers to specify access rights to MQTT topics. Based on the state of the authorization variables, the authorization manager assigns permissions to a connecting client. Authorization variables are usually manipulated in the authorization-preprocessing step of a pipeline.

Table 5. Authorization variables
Name Data type Description

authorization-key

String

The authorization-key is generally used by the authorization manager to assign client specific permissions.

authorization-role-key

String

The authorization-role-key is generally used by the authentication manager to assign permissions that are inherent to a conceptional group of clients.


Cryptographic Operations

Usage of Cryptography

The ESE uses cryptography to compare securely-stored secrets with the secrets that the MQTT clients provide for authentication. To make the comparison, the ESE currently employs four different password hashing schemata.

All of the schemata can be scaled in computational difficulty and all schemata allow for the use of an optional salt. Difficulty scaling is done with a difficulty parameter (if the one-way function that is used supports this parameter) or by using the output of the function as input in additional rounds of hashing. The Salt is used in the way that is specified by the one-way function or it is prepended once before the first round of hashing is done.

The ESE expects all byte strings such as hashed passwords to be encoded with the Base64 encoding scheme.

Once the cryptographic processing of the client-provided secret is done, the ESE compares the result with the configured trust for bit equality.

MD5

The ESE can use the MD5 message digest.

  • The salt is prepended to the secret before the first round of hashing.

  • The difficulty is scaled by using the output of MD5 again as input for iteration-times rounds.

  • The resulting hash has a length of 16 bytes or 24 characters in Base64 encoding.

MD5 is proven to be insufficiently strong for modern security standards and is only made available for testing and legacy purposes.

SHA512

The ESE can use the Secure Hashing Algorithm 2 with an output bit-length of 512 (SHA512).

  • The salt is prepended to the secret before the first round of hashing.

  • The difficulty is scaled by using the output of SHA512 again as input for iteration-times rounds.

  • The resulting hash has a length of 64 bytes or 87 characters in Base64 encoding.

BCRYPT

The ESE can use the Blowfish cipher based password hashing scheme bcrypt.

  • The salt is hashed with MD5 to resize it before use. Because bcrypt requires the use of a salt, an empty string is hashed and used if no salt is configured.

  • The secret is hashed with SHA512 to resize it before use.

  • The iteration parameter is directly used as difficulty parameter of bcrypt. Only values between 4 and including 31 are allowed.

  • The resulting hash has a length of 24 bytes or 32 characters in Base64 encoding.

PKCS5S2

The ESE can use the PKCS 5 V2.0 Scheme 2 as described in RFC 2898.

  • The underlying one-way function is SHA512.

  • The difficulty and the salt are directly passed on to the scheme.

  • The resulting hash has a length of 32 bytes or 44 characters in Base64 encoding.

PLAIN

The PLAIN schema (avaialble in ESE version 1.1) is not a cryptographic operation. This schema simply prompts the ESE to do a plain-text secret comparison (without any transformation of the provided secret beforehand).

  • No one-way function is used.

  • Any difficulty parameter or salt is ignored.

  • The length of the resulting byte string or Base64 encoding is highly dependent on the secret that is used.

Only use the PLAIN schema if you have a specific use case that requires this schema. Because the performance of the PLAIN schema is very different from real hash functions, the schema is unsuitable even for testing purposes.

Authentication and Authorization

The key tasks of the Enterprise Security Extension are authentication and authorization of MQTT clients. For a detailed explanation of these concepts in the context of HiveMQ, see the HiveMQ Documentation.

Authentication Managers

Authentication verifies whether a person, device, or application is who they say they are.

In the ESE, authentication is done by authentication managers. An authentication manager is defined in a pipeline and handles the authentication processes of connecting MQTT clients by using the content of the authentication variables.

Authorization Managers

Authorization defines which actions a specific person, device, or application is allowed to perform. Authorization can be split up into the concepts of roles and permissions. A permission contains a set of actions. Users who are granted the permission are allowed to do the actions that are defined in the permission. Permissions can also be assigned to a role. This role can then be assigned to a set of users. Users who have a certain role receive all of the permissions that are assigned to the role.

In the ESE, authorization is done by authorization managers. An authorization manager is defined in a pipeline and handles the authorization processes of connecting MQTT clients by using the content of the authorization variables.

We highly recommend the use of roles to assign permissions rather than the assignment of user-specific permissions. The use of roles to manage permissions creates a clear authority structure on your system and can improve overall performance of the ESE.


SQL Databases

The HiveMQ Enterprise Security Extension supports different SQL databases as external sources of authentication and authorization data. In the ESE configuration file, you can use the abstract representation of SQL realms to define the connection to one or or more databases. SQL realms can be referenced in SQL authentication managers and / or SQL authorization managers. To enable the ESE to work with the stored information, the databases must adhere to the SQL structure that is described below. SQL realms and the associated mangers support access control for the HiveMQ Control Center. To use the access-control feature for your control center, you need to set up additional SQL tables.

The ESE supports the following SQL databases:

See the full list of supported database versions.


SQL Realm

To connect the HiveMQ Enterprise Security Extension to an SQL database, simply configure an SQL realm in the <realms> section of your ESE configuration. The SQL realm must contain the database connection information and have a unique name that the authentication and authorization managers can reference.

Example SQL realm configuration
<enterprise-security-extension>
    ...

    <realms>
        <sql-realm>
            <name>sql-realm-name</name>
            <enabled>true</enabled>
            <configuration>
                <db-type>POSTGRES</db-type>
                <db-name>hivemq</db-name>
                <db-host>hostname</db-host>
                <db-port>5432</db-port>
                <db-username>hivemq</db-username>
                <db-password>password</db-password>
            </configuration>
        </sql-realm>
    </realms>

    ...
</enterprise-security-extension>
Table 6. SQL relam parameters
Parameter Type Mandatory Description

db-type

SQL type

Specifies the database technology

db-name

String

Specifies the database within the DBMS

db-host

String

Specifies the remote address of the database

db-port

Integer

Specifies the remote port of the database

db-username

String

Specifies the user name that the ESE uses to connect to the database

db-password

String

Specifies the password that the ESE uses to authenticate itself

Caching

The querying of SQL databases incurs a significant overhead in execution time. To remedy this, ESE caches information gained from connected databases.

Each Type of data is cached for a certain period of time before a time-based eviction happens:

As cache size is limited, evictions may happen before the caching time expires.

These caching rules also apply to the corresponding control-center data.

Supported Database Technologies

Since the ESE supports multiple types of SQL databases, you must specify the type of database for each realm. Each supported database can be specified by a predefined string.

Table 7. SQL types for SQL realm configuration
SQL Type Database Technology

POSTGRES

Use this type to connect to a PostgreSQL database

MYSQL

Use this type to connect to a MySQL or MariaDB database

MSSQL

Use this type to connect to an MSSQL database

AURORA_POSTGRES

Use this type to connect to an Amazon Aurora database, using the PostgreSQL API.

AURORA_MYSQL

Use this type to connect to an Amazon Aurora database, using the MySQL API.

AZURE_SQL

Use this type to connect to an Azure SQL database

Amazon Aurora offers the possibility to use a PostgreSQL or a MySQL API. You can specify the preferred type at database creation. If you do not know which database type you are using, you can check the RDS overview in the AWS management console.


SQL Authentication Manager

To use an SQL database as an external source for authentication, configure an SQL authentication manager in the desired pipeline and reference the corresponding SQL realm name in the authentication manager.

The ESE looks up the authentication information in the database by using the authentication variables. By default, they are already filled with the MQTT variables mqtt-username and mqtt-password. If you want to adapt the values of the authentication variables, you can use a preprocessor before the SQL authentication manager.

If you configure preprocessors before the authentication manager, ensure to set the authentication variables, as the mqtt-username and mqtt-password will not be used automatically in this case.

To ensure that the ESE can find the authentication information in the database, the layout of the database must be in accordance with the database table 'users' of the SQL database structure. If the authentication manager gets also authorization information from the database, it sets the authorization variables authorization-key and authorization-role-key too.

Example SQL authentication manager configuration
<enterprise-security-extension>

    <!-- add realms configuration -->

    <pipelines>
        <pipeline>
            <!-- add authentication preprocessors if necessary -->

            <sql-authentication-manager>
                <realm>sql-realm-name</realm>
            </sql-authentication-manager>

            <!-- add authorization preprocessors if necessary -->
            <!-- add an authorization manager -->
        </pipeline>
    </pipelines>

</enterprise-security-extension>
Table 8. SQL authentication manager parameters
Parameter Type Mandatory Description

realm

String

References the SQL realm from which the authentication data is drawn.


SQL Authorization Manager

To use an SQL database as an external source for authorization, configure an SQL authorization manager in the desired pipeline and reference the corresponding SQL realm name in the authorization manager.

The authorization manager also lets you define whether if you want to use roles and / or user specific permissions. Based on the configuration, the ESE uses different authorization variables. These variables can be set and manipulated with a preprocessor before the authorization manager.

Table 9. SQL authorization manager parameters
Parameter Type Mandatory / Default Description

realm

String

References the SQL realm from which the authentication data is drawn.

use-authorization-key

Boolean

false

If this parameter is set to true, the ESE uses user-specific permissions.

use-authorization-role-key

Boolean

true

If this parameter is set to true, the ESE uses role permissions.

Using Role Permissions

If you want to use role permissions, set <use-authorization-role-key> to true. The variable authorization-role-key must be set, so that the ESE can query the corresponding role and its permissions from the database.

Example SQL authorization manager configuration using role permissions
<enterprise-security-extension>

    <!-- add realms configuration -->

    <pipelines>
        <pipeline>
            <!-- add authentication preprocessors if necessary -->
            <!-- add an authentication manager -->
            <!-- add authorization preprocessors if necessary -->

            <sql-authorization-manager>
                <realm>sql-realm-name</realm>
                <use-authorization-key>false</use-authorization-key>
                <use-authorization-role-key>true</use-authorization-role-key>
            </sql-authorization-manager>

        </pipeline>
    </pipelines>

</enterprise-security-extension>

To ensure that the ESE finds the authorization information in the database, the layout of the database must be in accordance with the following database tables from the SQL database structure:

We recommend that you use role permissions rather than of user-specific permissions.
Using User-specific Permissions

If you want to use user-specific permissions, set <use-authorization-key> to true. The variable authorization-key must be set, so that the ESE can query the corresponding user-specific permissions from the database.

Example SQL authorization manager configuration using user-specific permissions
<enterprise-security-extension>

    <!-- add realms configuration -->

    <pipelines>
        <pipeline>
            <!-- add  authentication preprocessors if necessary -->
            <!-- add  authentication manager -->
            <!-- add  authorization preprocessors if necessary -->

            <sql-authorization-manager>
                <realm>example-backend</realm>
                <use-authorization-key>true</use-authorization-key>
                <use-authorization-role-key>false</use-authorization-role-key>
            </sql-authorization-manager>

        </pipeline>
    </pipelines>

</enterprise-security-extension>

To ensure that the ESE finds the authorization information in the database, the layout of the database must be in accordance with the following database tables from the SQL database structure:


SQL Database Setup

SQL Driver Installation

Each SQL realm must create a JDBC database connection to the configured database instance. To create the connection, a database specific JDBC Driver is required.

Because the license requirements of the different JDBC driver implementations differ widely, the ESE does not come prepackaged with a driver. To provide the ESE with the necessary JDBC driver, simply drop the corresponding jar file into the <ESE-Home>/drivers/jdbc directory. Include all of the JDBC driver jar files that you need, so that the drivers can be loaded at extension start.

The following table offers links to drivers and additional information:

Table 10. JDBC drivers
Database Download Link Minimum Version

PostgreSQL and Amazon Aurora

Download link

JDBC 4.2

MySQL, MariaDB and Amazon Aurora

Download link

Connector/J 8.0

MSSQL and Azure SQL

Download link

JDB Driver 7.2

SQL Database Setup

For the ESE to work properly, the SQL database has to have a certain schema. This structure is described in detail below.

To facilitate database setup, the ESE provides an SQL setup snippet for every supported SQL database. This snippet creates a database with all the required tables and columns. The SQL setup snippet can be found in the <ESE-Home>/sql/<ESE-Version> folder as text file. By copying the content and inserting it into your database management system or using it directly as SQL script it generates the needed structure automatically.

Once the necessary tables are created, use the usual SQL commands to fill the tables with the data of your clients as desired.

To help you store the passwords in an appropriate format, the ESE provides a CLI tool. For example, use the CLI tool to create SQL INSERT statements for the 'users' table and ensure that all passwords are stored in a way that the ESE can process. Of course you can process passwords and store them in the database without the provided CLI tool. The cryptography section explains how the ESE processes passwords. Always ensure that the passwords are stored in a way so that the ESE can validate them.

If you already have an supported SQL database, simply verify that your data is stored in a way that the ESE can work with. If you do not have a matching SQL structure, this SQL view documentation shows a way to create a database view that is in accordance with the required database structure of the ESE.

SQL Database Structure

Entity Relation Diagram

If you use the provided sql scripts to set up your database in version 1.2 or later, two additional columns are added to each table: created_at and updated_at. These columns are of the appropriate database-specific data type for storing an uniquely identifiable timestamp. The entry created_at is always set to the time the database record was created. The entry updated_at is always set to the time of the last change to the database record. There is no need for you to interact with either created_at or updated_at manually.
Table 11. Table "users"
Column Data Type Description

username

Text

Primary Key, Unique, Not Null

password

Text

Base64 Encoded raw byte array

password_iterations

Integer

Not Null

password_salt

Text

Not Null

algorithm

Text

Not Null

Table 12. Table "roles"
Column Data Type Description

id

Serial

Primary Key, Unique, Not Null

name

Text

Unique, Not Null

description

Text

Table 13. Table "user_roles"
Column Data Type Description

user_id

Integer

Primary Key

role_id

Integer

Primary Key

Table 14. Table "permissions"
Column Data Type Description

id

Serial

Primary Key

topic

Text

Not Null

publish_allowed

Boolean

Default false, Not Null

subscribe-allowed

Boolean

Default false, Not Null

qos_0_allowed

Boolean

Default false, Not Null

qos_1_allowed

Boolean

Default false, Not Null

qos_2_allowed

Boolean

Default false, Not Null

retained_msgs_allowed

Boolean

Default false, Not Null

shared_sub_allowed

Boolean

Default false, Not Null

shared_group

Text

Table 15. Table "role_permissions"
Column Data type Description

role

Integer

Primary Key, Not Null

permission

Integer

Primary Key, Not Null

Table 16. Table "user_permissions"
Column Data Type Description

user_id

Integer

Primary Key, Not Null

permission

Integer

Primary Key, Not Null

SQL Database Structure for Control Center Access Control

The ESE-access control feature for your control center requires additional tables in your SQL database. If you only want to authenticate or authorize MQTT clients, no additional tables are required.

Entity Relation Diagram

If you use the provided sql scripts to set up your database in version 1.2 or later, two additional columns are added to each table: created_at and updated_at. These columns are of the appropriate database-specific data type for storing an uniquely identifiable timestamp. The entry created_at is always set to the time the database record was created. The entry updated_at is always set to the time of the last change to the database record. There is no need for you to interact with either created_at or updated_at manually.
Table 17. Table "cc_users"
Column Data Type Description

username

Text

Primary Key, Unique, Not Null

password

Text

Base64 Encoded raw byte array

password_iterations

Integer

Not Null

password_salt

Text

Not Null

algorithm

Text

Not Null

Table 18. Table "cc_roles"
Column Data Type Description

id

Serial

Primary Key, Unique, Not Null

name

Text

Unique, Not Null

description

Text

Table 19. Table "cc_user_roles"
Column Data Type Description

user_id

Integer

Primary Key

role_id

Integer

Primary Key

Table 20. Table "cc_permissions"
Column Data Type Description

id

Serial

Primary Key

topic_string

Text

Not Null

description

Text

A script to insert all supported HiveMQ permissions is provided in the sql folder or you can just copy paste the one below.

SQL INSERT HiveMQ Control-Center Permissions
-- insert the default permissions strings that are supported by HiveMQ into the cc_permissions
insert into cc_permissions (permission_string, description)
    values  ('HIVEMQ_SUPER_ADMIN', 'special cc_permission, that allows access to everything'),
            ('HIVEMQ_VIEW_PAGE_CLIENT_LIST', 'allowed to view client list'),
            ('HIVEMQ_VIEW_PAGE_CLIENT_DETAIL', 'allowed to view client detail'),
            ('HIVEMQ_VIEW_PAGE_LICENSE', 'allowed to view license page'),
            ('HIVEMQ_VIEW_PAGE_TRACE_RECORDINGS', 'allowed to view trace recording page'),
            ('HIVEMQ_VIEW_PAGE_DROPPED_MESSAGES', 'allowed to dropped message page'),
            ('HIVEMQ_VIEW_PAGE_BACKUP', 'allowed to view backup page'),
            ('HIVEMQ_VIEW_DATA_CLIENTID', 'allowed to see client identifiers'),
            ('HIVEMQ_VIEW_DATA_IP', 'allowed to see client''s IPs'),
            ('HIVEMQ_VIEW_DATA_PAYLOAD', 'allowed to see message payloads'),
            ('HIVEMQ_VIEW_DATA_PASSWORD', 'allowed to see client''s passwords'),
            ('HIVEMQ_VIEW_DATA_USERNAME', 'allowed to see client''s usernames'),
            ('HIVEMQ_VIEW_DATA_WILL_MESSAGE', 'allowed to see client''s LWT'),
            ('HIVEMQ_VIEW_DATA_TOPIC', 'allowed to see client''s topics'),
            ('HIVEMQ_VIEW_DATA_SUBSCRIPTION', 'allowed to see client''s subscriptions'),
            ('HIVEMQ_VIEW_DATA_PROXY', 'allowed to see client''s proxy information'),
            ('HIVEMQ_VIEW_DATA_TLS', 'allowed to see client''s TLS information')
;
Table 21. Table "cc_role_permissions"
Column Data type Description

role

Integer

Primary Key, Not Null

permission

Integer

Primary Key, Not Null

Table 22. Table "user_permissions"
Column Data Type Description

user_id

Integer

Primary Key, Not Null

permission

Integer

Primary Key, Not Null

Supported SQL database versions
Table 23. Supported SQL database versions
SQL Database Supported Versions

MySQL

  • 5.6

  • 5.7

  • 8.0

MSSQL

  • 2017-CU12

  • 2018-CTP20

  • 2019-CTP3.1

PostgreSQL

  • 9.4

  • 9.5

  • 9.6

  • 10

  • 11

  • 12

MariaDB

  • 10.1

  • 10.2

  • 10.3

  • 10.4

Amazon Aurora

  • Amazon Aurora with MySQL compatibility (see our supported versions for MySQL)

  • Amazon Aurora with PostgreSQL compatibility (see our supported versions for PostgreSQL)

Azure SQL

  • See our supported versions for MSSQL


Preprocessing

In a pipeline, you can add preprocessing steps before authentication and before authorization. The preprocessing is done through the configuration of one or more preprocessors. Preprocessors are lightweight pipeline steps that work with the current state of the ESE variables and have no external dependencies.


Plain Preprocessor

The plain preprocessor offers a way to transfer the state of ESE variables. It contains a list of one or more transformations. During runtime, the transformations are processed in the order in which they appear in the configuration file.

Plain preprocessor configuration
<plain-preprocessor>
    <transformations>
        <transformation>...</transformation>
        <transformation>...</transformation>
        <transformation>...</transformation>
        <!-- add  more transformations-->
    </transformations>
</plain-preprocessor>

A transformation copies values from one ESE variable to another ESE variable. These variables are annotated as from and to variables. After the transformation process, the value in the from variable is stored in the to variable as well as in the from variable. Constant ESE variables are not permitted in a to variable.

If the data type of the from and to variables differ, an encoding must be specified.

Transformation configuration
<transformation encoding="UTF8">
    <from>byte-1</from>
    <to>string-1</to>
</transformation>
Table 24. Transformation parameters
Parameter Type Description

encoding

Encoding

Defines how the ESE variable referenced in the from tag is interpreted. This parameter is only valid if the from and to variables have different data types and require a type-conversion.

from

String

References the ESE variable in which the incoming data of the transformation is stored.

to

String

References the ESE variable in which the outgoing data of the transformation is stored.

Table 25. Transformation encodings
Encoding Description

UTF8

Uses the UTF-8 encoding as specified in RFC 3629.

Base64

Uses the Base64 encoding as specified in RFC 4648. Some transformations produce or require a padding of up to two = characters.


Regex Preprocessor

The regex preprocessor allows you to pull substrings from a string ESE variable with the help of regular expressions. The preprocessor takes the value of the selected string variable, matches a pattern, and writes the resulting matches back into the string variable.

The regex preprocessor is useful when just a part of an ESE variable must be used for authentication or authorization purposes. Here are some examples:

  • All client passwords start with the phrase password followed by the real secret.

  • A part of the client ID needs to be inserted into the topic permissions to specify the client.

  • Only the first three characters of the user name are relevant.

The regex preprocessor contains a regular expression pattern, an optional group number, and a list of ESE variables to which the preprocessor is applied.

Regex preprocessor configuration
<regex-preprocessor>
    <pattern>mqtt-client=((.{3})-(.{3}))</pattern>
    <group>1</group>
    <variable>authentication-key</variable>
</regex-preprocessor>
Table 26. Regex preprocessor parameters
Parameter Type Description

pattern

String

The regular expression that is used

group

Positive integer or 0

The group in the regular expression that is extracted. Groups are framed with parenthesis ( and ). The 0 group is always the whole pattern. The default is 0.

variable

String

References the ESE variable that is processed. This tag can be present multiple times.

Regular Expression Syntax

The regex preprocessor expands the regular expressions syntax that the HiveMQ Control Center uses to include grouping logic.

To add a group to a pattern, include a pair of matching parenthesis (). Groups are ordered by their opening parenthesis ( from left to right. Regardless of the parenthesis, the 0 group is always present and contains the whole matched pattern.

For example, a variable that contains the string "SOMETHINGmqtt-client=abc-defSOMETHING" is processed with the pattern mqtt-client=((.{3})-(.{3})) that contains four groups. Based on the matched group, the value of the string variable after preprocessing is as follows:

Table 27. Regular expression group example
Group Number New Value

0

mqtt-client=abc-def

1

abc-def

2

abc

3

def


Permission Placeholders

Permission placeholders facilitate the creation of permissions for groups of clients that require similar, but client specific permissions. The use of permission placeholders can eliminate the need to create individual permissions for each client.

The placeholders can be used inside the topic filter and shared group of a permission. All ESE variables of the string data type can be used as placeholders. At the end of the authorization step, the placeholders are dynamically replaced with the values of the referenced ESE variables.

To create a permission with a placeholder, insert the name of the ESE variable framed with ${ and } at the desired position in the permission.

Example for permission placeholders:

  • The authorization manager grants subscription to the topic filter topic/${mqtt-clientid}/${string-1}/subtopic.

  • If the value of mqtt-clientid is my-special-client and string-1 is last, then the connecting client is only allowed to subscribe with the topic filter topic/my-special-client/last/subtopic.

Only variables of the string data type can be substituted. Byte variables can not be replaced.


Access Log

The access log provides a single, unified log for tracking security-relevant data. This log has several use cases:

  • You can audit all accesses that the ESE grants retroactively.

  • Your intrusion-prevention software (for example, Fail2Ban) can use the access log to create firewall rules.

  • The chronological records of the access log can provide valuable information for the post-mortem of a data breach.

The access log shows you precisely who was granted access to HiveMQ, what kind of access occurred, and when the access was granted. The timezone of events in the access log is always Coordinated Universal Time (UTC).

A good start for a fail-regex may be this: ^ - authentication-failed - Client failed authentication: ID [^\s]+, IP <HOST>, reason \“((authorization timed out)|(unknown authentication key or wrong authentication secret)|(authentication timed out)|(other))\“\.$


Access Events

The three primary access events are Authentication Failed, Authentication Succeeded, and Authorization Succeeded. Each of these events provides the public client IP address and the MQTT client ID of the connecting client.

Table 28. Access log events
Event Type Logged Values Log Statement

Authentication Failed

  • Client ID

  • Client IP

  • Reason of failure

yyyy-MM-dd hh:mm:ss,sss UTC - authentication-failed - Client failed authentication: ID <client id>, IP <public client ip>, reason "<reason string>"

Authentication Succeeded

  • Client ID

  • Client IP

yyyy-MM-dd hh:mm:ss,sss UTC - access.authentication-succeeded - Client succeeded authentication: ID <client id>, IP <public client ip>.

Authorization Succeeded

  • Client ID

  • Client IP

  • List of permissions

yyyy-MM-dd hh:mm:ss,sss UTC - access.authorization-succeeded - Client succeeded authorization: ID <client id>, IP <public client ip>, permissions <[permission]>.

Authentication Failed

This event triggers when an MQTT client does not pass the ESE authentication and is not allowed to connect to HiveMQ. Each Authentication Failed event provides one of the following reasons for the failure:

Table 29. Authentication failed reasons
Reason Description Corresponding Metrics

unknown authentication key or wrong authentication secret

The client cannot be authenticated against the information that the configured realm of the authentication manager provided

unknown-key, wrong-secret

authentication timed out

The process timed out during the authentication step of the pipeline

authentication-timeout

authorization timed out

The process timed out during the authorization step of the pipeline

authorization-timeout

other

Something exceptional happened during the process. In this case, you can look for more information in the DEBUG output of the hivemq.log.

other

Authentication Succeeded

This event triggers when an MQTT client passes the ESE authentication successfully. These events correspond to the authentication-succeeded metric.

Authorization Succeeded

This event triggers after an MQTT client is granted permissions. The event references the Topic Permissions that the client is granted in the following format:

Permission{topicFilter='<topic-string>', qos=[<0, 1, 2>], activity=[<publish, subscribe>], retainedPublishAllowed=<true|false>, sharedSubscribeAllowed=<true|false>, sharedGroup='<group-id>', from='<source>'}.

The <source> in the reference is dependent on the type of authorization manger used. For an SQL authorization manager, the source can either be the roles.name or the users.username that the permission is associated with.


Access Log File

By default, the access log writes to the <HiveMQ Home>/log/access/access.log. You can customize this path in the ESE configuration file.

The access log provides automatic log file rolling. Once per day at midnight, the old access log files are compressed with the gzip algorithm and archived with the filename access.<yyyy-MM-dd>.log.gz. For example, after two days of operation the access folder contains the following files:

├─ access.2019-07-04.log.gz
├─ access.2019-07-05.log.gz
└─ access.log
The ESE never deletes the access log files that are archived. If you need to remove old access logs regularly, you must take additional action. For example, set up a scheduled cron job to alleviate data protection concerns or storage constraints.


Access Log Configuration

In the enterprise-security-extension.xml file, you can use the optional <access-log> tag to configure the behavior of the access log.

Implicit default access log configuration
<access-log>
    <enabled>true</enabled>
    <file-name>access</file-name>
    <sub-folder>access</sub-folder>
    <write-through>false</write-through>
</access-log>
Table 30. Access log configuration parameters
Parameter Type Description Default

enabled

Boolean

Enables or disables writing access events to the access log file

true

file-name

String

The name of the access log file, a .log is suffixed. This changes the archived log file names as well.

access

sub-folder

String

The name of the subfolder in the log directory of HiveMQ where the access log file and the archived log files are placed

access

write-through

Boolean

Enables or disables also writing access events to the hivemq.log file. The log level of the access events is INFO. If you want to write the access events only to the hivemq.log file, set the enabled tag to false and write-through to true.

false


Control Center Access Control

The ESE provides access control for the HiveMQ Control Center. It is possible to use both role based access control (RBAC) and finer grained access control with user permissions.

Requirements for Control Center Access Control

  • HiveMQ Enterprise Security Extension 1.2.0 or higher

  • HiveMQ Enterprise Edition 4.2.0 or higher

Control Center Pipeline

Similar to the Authentication and Authorization of MQTT clients, the handling of Control Center users is done in a separate pipeline: the <control-center-pipeline>. This special ESE pipeline works like a listener pipeline for MQTT clients, but it omits authentication and authorization preprocessing and does not require transformation of ESE variables.

The control center pipeline uses the User form of the login page as the authentication-key and authorization-key. After the Password form is decoded with the UTF8 codec, this information is used as the authentication-byte-secret.

Control center login page

HiveMQ control center login page

Control center pipeline configuration
<enterprise-security-extension>
    <pipelines>
        <!-- only one <control-center-pipeline> allowed -->
        <!-- secure access to the control center -->
        <control-center-pipeline>
            <!-- authenticate over a sql db -->
            <sql-authentication-manager>
                <realm>postgres-backend</realm>
            </sql-authentication-manager>
            <!-- authorize over a sql db -->
            <sql-authorization-manager>
                <realm>postgres-backend</realm>
                <use-authorization-key>true</use-authorization-key>
                <use-authorization-role-key>true</use-authorization-role-key>
            </sql-authorization-manager>
        </control-center-pipeline>
    </pipelines>
</enterprise-security-extension>
Refer to the description of each realm to learn how to enable access control for the control-center from the realm.

Control Center Permissions

To implement fine-grained access control for your control center, you assign users specific control-center permissions. If desired, you can assign a set of control-center permissions to a role. All users with a specific role are granted the permissions that are assigned to the role. Role-based access control can simplify permission management and increase performance.

Below are the available permissions for the HiveMQ control-center:

Table 31. Control center permissions
Permission Description

HIVEMQ_SUPER_ADMIN

Allows the user to access everything in your control center. This special permission is the default role for users in the HiveMQ config.xml.

HIVEMQ_VIEW_PAGE_CLIENT_LIST

Allows the user to view the client list on your control center

HIVEMQ_VIEW_PAGE_CLIENT_DETAIL

Allows the user to view client details on your control center

HIVEMQ_VIEW_PAGE_LICENSE

Allows the user to view the license information on your control center

HIVEMQ_VIEW_PAGE_TRACE_RECORDINGS

Allows the user to view trace recordings on your control center

HIVEMQ_VIEW_PAGE_DROPPED_MESSAGES

Allows the user to view dropped messages on your control center

HIVEMQ_VIEW_PAGE_BACKUP

Allows the user to view backup information on your control center

HIVEMQ_VIEW_DATA_CLIENTID

Allows the user to view client identifiers on your control center

Lack of this permission has the following result:

  • The user cannot view the Clients, Analytics, or Trace Recordings pages of your control center

  • Client identifiers are hidden from the user on all pages of your control center

HIVEMQ_VIEW_DATA_IP

Allows the user to view client IP information in your control center

Lack of this permission has the following result:

  • The user cannot view the Trace Recordings page of your control center

  • The IP information of clients is hidden from the user on all pages of your control center.

HIVEMQ_VIEW_DATA_PAYLOAD

Allows the user to view message payloads on your control center

Lack of this permission has the following result:

  • The user cannot view the Trace Recordings page in your control center

  • Message payloads are hidden from the user on all pages of your control center

HIVEMQ_VIEW_DATA_PASSWORD

Allows the user to view client passwords in your control center

Lack of this permission has the following result:

  • The user cannot view the Trace Recordings page of your control center

  • Client passwords are hidden from the user on all pages of your control center

HIVEMQ_VIEW_DATA_USERNAME

Allowed to see client’s usernames.

Lack of this permission has the following result:

  • The user cannot view the Trace Recording page of your control center

  • User name information of clients is hidden from the user on all pages of your control center

HIVEMQ_VIEW_DATA_WILL_MESSAGE

Allows the user to view the LWT of clients on your control center

Lack of this permission has the following result:

  • The user cannot view the Trace Recording page of your control center

  • LWT information is hidden from the user on all pages of your control center

HIVEMQ_VIEW_DATA_TOPIC

Allows the user to view the PUBLISH and SUBSCRIBE topics of clients on your control center

Lack of this permission has the following result:

  • The user cannot view the Trace Recordings page of your control center

  • Topic information is hidden from the user on all pages of your control center

HIVEMQ_VIEW_DATA_SUBSCRIPTION

Allows the user to see client subscriptions on your control center

Lack of this permission has the following result:

  • The user cannot view the Trace Recordings page of your control center

  • Subscription information is hidden from the user on all pages of your control center

HIVEMQ_VIEW_DATA_PROXY

Allows the user to view the proxy information of clients on your control center

Lack of this permission has the following result:

  • The user cannot view the Trace Recordings page of your control center

  • Client-proxy information is hidden from the user on all pages of your control center

HIVEMQ_VIEW_DATA_TLS

Allows the user to view the TLS information of clients on your control center

Lack of this permission has the following result:

  • TLS information is hidden from the user on all pages of the control center


Monitoring

The ESE exposes several metrics over the standard HiveMQ monitoring interface.

ESE metrics always start with the prefix com.hivemq.extensions.ese. The following table lists all of the metrics that are exposed. For increased readability, prefixes are omitted. For more information on HiveMQ metrics, see metric types.

Table 32. ESE Metrics
Metric (without prefix) Type Description

connect

Counter

Shows the total number of incoming MQTT client connection attempts that the ESE processes

access.authentication-succeeded

Counter

Shows the total number of successful authentication requests

access.authorization-succeeded

Counter

Shows the total number of successful authorization requests

access.authentication-failed

Counter

Shows the total number of unsuccessful authentication requests

access.authentication-failed.unknown-key

Counter

Shows the total number of authentication requests that failed due to unknown authentication keys

access.authentication-failed.wrong-secret

Counter

Shows the total number of authentication requests that failed due to incorrect authentication secrets

access.authentication-failed.authentication-timeout

Counter

Shows the total number of authentication requests that failed due to timed-out authentications

access.authentication-failed.authorization-timeout

Counter

Shows the total number of authentication requests that failed due to timed-out authorizations

access.authentication-failed.other

Counter

Shows the total number of authentication requests that failed due to other reasons

Each SQL realm has its own metrics. The name of the realm is included in the metric name.

Table 33. ESE SQL Metrics
Metric (without prefix) Type Description

sql.permission.role.cache.hitrate.<realm_name>

Meter

Shows the rate of requests for role-based permission data that are handled from the ESE cache. The rate is provided in requests per a pre-defined time interval for the specified SQL realm.

sql.permission.user.cache.hitrate.<realm_name>

Meter

Shows the rate of requests for user-based permission data that are handled from the ESE cache. The rate is provided in requests per a pre-defined time interval for the specified SQL realm.

sql.permission.time.<realm_name>

Timer

Shows the duration and frequency of requests for authorization data from the specified SQL realm

sql.user.time.<realm_name>

Timer

Shows the duration and frequency of requests for authentication data from the specified SQL realm


Support

If you need help with the HiveMQ Enterprise Security Extension or have suggestions on how we can improve the extension, please contact us at contact@hivemq.com.