What’s New in HiveMQ 4.23?

The HiveMQ team is proud to announce the release of HiveMQ Enterprise MQTT Platform 4.23. This release focuses on new ways to harness the power of IoT device data with HiveMQ.

Highlights

• New HiveMQ Enterprise Data Lake Extension
• HiveMQ Data Hub Transformations with Custom Scripting
• HiveMQ Enterprise Security Extension Enhancements

HiveMQ Enterprise Data Lake Extension

This month, we are excited to introduce another great HiveMQ platform extension: the HiveMQ Enterprise Data Lake Extension. This new addition to the HiveMQ platform bundle lets you store MQTT messages for easy integration into the AWS-based data lake architecture of your choice.

How it works

Data lakes are centralized repositories that allow organizations to store vast amounts of data at scale. Unlike traditional databases that often require specific data types and structured information, data lakes are designed to handle a wide range of data types, including raw, semi-structured, and unstructured data.

The HiveMQ Enterprise Data Lake Extension helps you optimize the storage of your IoT device data on AWS S3 and seamlessly integrates MQTT data into the data lake of your choice for further processing and analysis.

The extension stores MQTT data from IoT devices in Parquet file format on Amazon S3 for ingestion into data lake solutions, such as Data Lake on AWS, DataBricks, and Snowflake. The extension writes Apache Iceberg-compatible Parquet files.

Example Data Lake Extension configuration file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

<hivemq-data-lake-extension>
  <aws-credential-profiles>
    <aws-credential-profile>
      <id>my-credentials-id</id>
      <profile-file>/opt/examples/aws-credentials</profile-file>
  </aws-credential-profile>
</aws-credential-profiles>
<mqtt-to-s3-routes>
  <mqtt-to-s3-route>
    <id>my-mqtt-to-s3-route-id</id>
    <mqtt-topic-filters>
        <mqtt-topic-filter>#</mqtt-topic-filter>
    </mqtt-topic-filters>
    <aws-credential-profile-id>my-credentials-id</aws-credential-profile-id>
    <bucket>my-bucket</bucket>
    <region>eu-central-1</region>
    <processor>
      <parquet>
        <columns>
            <column>
              <name>topic</name>
              <value>mqtt-topic</value>
            </column>
            <column>
              <name>payload</name>
              <value>mqtt-payload</value>
            </column>
            <column>
              <name>timestamp</name>
              <value>timestamp</value>
            </column>
            </columns>
          </parquet>
        </processor>
      </mqtt-to-s3-route>
    </mqtt-to-s3-routes>
  </hivemq-data-lake-extension>

The HiveMQ Enterprise Data Lake Extension provides predefined MQTT placeholders for converting MQTT messages into Parquet table rows to ensure data is stored in your cloud storage according to your business needs. For a list of all available placeholders, visit our Data Lake Extension documentation.

How it helps

The ability of data lakes to store data in its native format and process any amount or variety of data can help organizations integrate and analyze data from different parts of the business. The HiveMQ Enterprise Data Lake Extension ensures the smooth and efficient transfer of IoT data to your cloud-based data lake solution. The data lake solution can be configured to extract and ingest IoT data from cloud storage to enable data analysis, data engineering, machine learning, and more.

HiveMQ Data Hub Transformations with Custom Scripting (EAP)

HiveMQ 4.23 launches the Early Access Preview (EAP) of the Data Hub transformation feature. Data Hub transformations allow users to directly transform MQTT payloads in flight using custom JavaScript functions.

NOTE: The Data Hub Transformations feature is an EAP. In this release candidate, Linux arm64 is not yet supported.

How it works

JavaScript is a powerful and widely used programming language that has become a staple in the world of software development. The extensive standardization JavaScript offers has made it the go-to language for developers to iterate applications quickly. Data Hub now provides an easy-to-use JavaScript-flavored interface to transform MQTT payloads.

The following function depicts the transformation of a JSON MQTT message payload from a temperature sensor. The sensor provides temperature data in Fahrenheit, but the consumer, a fan, requires the data to be in Celsius.

Example Data Hub function with a transformation script:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

  function convert(fahrenheit) {
      return Mah.floor((fahrenheit - 32) * 5/9);
  }

  function transform(publish, context) {
      publish.payload = {
        "celsius": convert(publish.payload.fahrenheit),
       "timestamp": publish.payload.timestamp
      }
      return publish;
  }

Every script must have a transform function that serves as the entry point for all configured transformation functions. Data Hub invokes the JavaScript function and passes the MQTT payload and context object. The returned MQTT message payload undergoes further processing within the broker.

For more information, see Data Hub Transformations.

How it helps

IoT deployments typically include a variety of data sources and generate a diverse range of data. To use this data effectively, data transformations of some kind can be necessary. Rather than managing additional and complex software components, the desired transformations can now be executed in HiveMQ Data Hub while your data is in flight. Our effective new transformation feature will allow developers to iterate even faster on their IoT use cases. Data Hub offers an easy-to-use interface between MQTT messages and JavaScript to transform them easily.

ESE Client Certificate Revocation Checking

HiveMQ 4.23 introduces support for TLS handshake client certificate revocation checking with OCSP (Online Certificate Status Protocol) and CRL (Certificate Revocation List) for TLS listeners to the HiveMQ Enterprise Security Extension.

How it works

Now, you can configure one or more revocation checks of your choice in the ESE configuration and reference TLS listener configurations in the broker. When a client/device connects to the HiveMQ broker, certificates are checked against a list of revoked certificates. If the client certificate has been revoked the connection is refused.

Example ESE configuration of an OCSP revocation check:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

<enterprise-security-extension>
    <client-certificate-revocation-checks>
        <client-certificate-revocation-check listener="my-listener">
            <on-tls-handshake>
                <method>
                    <ocsp/>
                </method>
            </on-tls-handshake>
        </client-certificate-revocation-check>
    </client-certificate-revocation-checks>
    ...
</enterprise-security-extension>

Example HiveMQ broker configuration of a TLS TCP listener:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

<hivemq>
    <listeners>
        <tls-tcp-listener>
            <name>my-listener</name>
            <tls>
                <keystore>...</keystore>
                <truststore>...</truststore>
                <client-authentication-mode>REQUIRED</client-authentication-mode>
                <session-timeout>300</session-timeout>
            </tls>
            ...
        </tls-tcp-listener>
    </listeners>
    ...
</hivemq>

For further configuration information, see Client Certificate Revocation Check.

How it helps

IoT deployments that use certificates to secure client-to-broker communication sometimes need to revoke an existing client certificate before its natural expiration date. Revocation can be achieved with a certificate revocation list (CRL) or the online certificate status protocol (OCSP). In both cases, the client/device certificates are checked against a provided list of revoked certificates. The connection attempt is denied if the client/device certificate has been revoked. This process can increase the security for use cases such as connected car applications where being able to centrally manage access control for vehicles can be a safety and compliance concern.

ESE OIDC mutual TLS and private key JWT client authentication for token endpoints

Starting with HiveMQ 4.23, the HiveMQ Enterprise Security Extension adds support for mutual TLS and private key JWT client authentication for the OIDC (OpenID Connect) token endpoints to improve the security of token retrieval for the Control Center. This method increases security when connecting to the identity provider for token retrieval.

How it works

Now, you have the option to configure the authentication method of the OIDC realm in the ESE configuration. In addition to the previously available simple authentication, you can now use mutual TLS and private key JWT authentication.

Example ESE configuration for mutual TLS authentication:

1
2
3
4
5

<tls-client-auth>
    <keystore>
        <path>/my-keystore.jks</path>
    </keystore>
</tls-client-auth>

Example ESE configuration of private key JWT authentication:

1
2
3
4
5
6
7

<private-key-jwt>
    <keystore-entry-alias>my-private-key-alias</keystore-entry-alias>
    <keystore>
        <path>/my-keystore.jks</path>
    </keystore>
  <kid>my-kid</kid>
</private-key-jwt>

Check the configuration options of your identity provider to verify which authentication methods are supported. To learn more about authentication in the HiveMQ Security Enterprise Extension, see Authentication and Authorization.

How it helps

Depending on your use case, you can now improve security by using one of the new authentication methods when connecting to your identity provider for retrieving tokens.

Additional Features and Improvements

HiveMQ Enterprise MQTT Broker

• Provide multi-arch (amd64 and arm64/v8) container images for HiveMQ Platform version 4.23 on DockerHub.
• Added the ability to configure the JMX port that HiveMQ run scripts use with a new HIVEMQ_JMX_PORT environment variable.
• Fixed an issue that could interfere with error handling during a rolling upgrade in certain rare cases.
• Streamlined the way array allocation is handled for logging purposes to minimize CPU usage.

HiveMQ Data Hub

• Fixed an issue where the applyPolicies argument of the Delivery.redirect function was of type String instead of Boolean when a policy was created via the Control Center.

HiveMQ Enterprise Security Extension

• Improved the performance of LDAP authorization managers that use MQTT role permissions.

HiveMQ Enterprise Extension for MySQL

• Added the ability to map all MQTT 5 user properties of a PUBLISH message to a single MySQL column as a JSON string array.
• Added the ability to map each individual MQTT 5 user property of a PUBLISH message to a separate MySQL column as a string.

HiveMQ Swarm

• Provide multi-arch (amd64 and arm64/v8) container image for HiveMQ Swarm on DockerHub.

Get Started Today

To upgrade to HiveMQ 4.23 from a previous HiveMQ version, take a look at our HiveMQ Upgrade Guide. To learn more about all the features we offer, explore the HiveMQ User Guide.

About HiveMQ Team

We love writing about MQTT, IoT protocols and architecture in general. Our experts are here to help, so reach out to us if we can help!

Contact HiveMQ