MQTT Data Integration with AWS — HiveMQ's Amazon Kinesis Extension

HiveMQ has revolutionized the Internet of Things (IoT) space by becoming the central nervous system for IoT environments. With its full-featured MQTT broker, HiveMQ can connect to hyper scalers such as AWS, Azure Cloud, Kafka, and Google Cloud for seamless, bidirectional data transmission between IoT devices and these cloud solutions.

The Enterprise Extension for Amazon Kinesis, HiveMQ’s latest addition to its data integration solutions, provides additional flexibility for customers to fully integrate their IoT architectures with AWS cloud services. With the extension, HiveMQ can transform MQTT data into Kinesis records for easy bi-directional data integration with AWS and IoT devices.

Download the HiveMQ platform and test the Enterprise Extension for Amazon Kinesis into it’s platform. 

About the Enterprise Extension for Amazon Kinesis

This extension provides additional flexibility to integrate IoT architectures with AWS cloud services fully. With the 4.13 platform release, the extension enables seamless bidirectional MQTT data transmission between IoT devices and Amazon Kinesis Data Streams, via the HiveMQ platform.

Amazon Kinesis is a suite of fully-managed Amazon Web Services (AWS) designed to ingest, process, and analyze data streams. Amazon Kinesis Data Streams collects data from various sources in the Kinesis suite.

The extension features Message Fan-Out as well —it can be configured to forward or consume messages to one or more Kinesis data streams.

With the 4.14 release, the Enterprise Extension for Amazon Kinesis also features a powerful Customization SDK, allowing granular control over message transformation and bidirectional routing. MQTT messages can be processed and written to Kinesis, while Kinesis records are transported as MQTT messages to MQTT-enabled devices. This allows the implementation of your specific IoT use cases and architectures.

Learn more about HiveMQ’s solution for MQTT Data Integration with AWS.

Use-cases

Consider Amazon Kinesis Data Streams as an example. Many services on AWS cloud provide integration with Amazon Kinesis Data Streams. Kinesis acts as a message queue between producers and consumers of data. MQTT messages are delivered from the HiveMQ broker to Amazon Kinesis using HiveMQ’s Kinesis extension. These messages are now available for backend processing and storage on AWS. Amazon Kinesis Data Analytics, for example, can consume and process these messages for further storage to AWS S3, AWS Timestream, or other AWS storage services.

The specific cloud service or backend used will depend on your architecture needs and requirement. It may be on Amazon, Google, Azure, or your private on-premise cloud.

Get Started with the Amazon Kinesis Extension

HiveMQ provides a free trial version of the HiveMQ platform for download. The current version, HiveMQ 4.14, includes the full-featured HiveMQ Enterprise Extension for Amazon Kinesis.

The Enterprise Extension for Amazon Kinesis provides bidirectional movement of MQTT data between HiveMQ and Amazon Kinesis Data Streams. For each direction, you can configure two processors to handle the specifics of the data movement.

• The mapping processor, which provides configuration based standard data movement.
• The transformer processor, which provides a powerful customization SDK and full programmatic access to transform messages and the specifications of the data movement.

It is easy to install and to start. You can also reference the documentation to see how simple it is. Also, refer to the documentation for a detailed explanation of the configuration options and instructions on implementing your custom transformer.

Here is an example of a minimal configuration to move MQTT messages to your configured Amazon Kinesis Data Stream that uses the mapping processor:

<hivemq-amazon-kinesis-extension>
    <aws-credential-profiles>
        <aws-credential-profile>
            <id>profile-01</id>
            <profile-file>aws-credentials-file</profile-file>
        </aws-credential-profile>
    </aws-credential-profiles>

    <mqtt-to-kinesis-routes>
        <mqtt-to-kinesis-route>
            <id>my-mqtt-to-kinesis-route</id>
            <enabled>true</enabled>
            <aws-credential-profile-id>profile-01</aws-credential-profile-id>
            <region>eu-central-1</region>
            <mqtt-topic-filters>
                <mqtt-topic-filter>mqtt/topic/in</mqtt-topic-filter>
            </mqtt-topic-filters>
            <processor>
                <mapping>
                    <kinesis-streams>
                        <kinesis-stream>
                            <name>my-kinesis-stream</name>
                            <partition-key>
                                <mqtt-topic/>
                            </partition-key>
                            <explicit-hash-key>
                                <random/>
                            </explicit-hash-key>
                        </kinesis-stream>
                    </kinesis-streams>
                </mapping>
            </processor>
        </mqtt-to-kinesis-route>
    </mqtt-to-kinesis-routes>
</hivemq-amazon-kinesis-extension>

To use the transformer processor, we provide an extensive Customization SDK that provides full programmatic access to the transformation process of a message. The Customization SDK documentation includes many examples as well.

In addition, HiveMQ provides an example, “Hello World Customization” for Amazon Kinesis on Github, for your convenience to get started with writing your transformers.

HiveMQ Enables a Multi-cloud Strategy for IoT

Message queue systems such as Amazon Kinesis, Google Pub/Sub, Kafka, and Azure EventHubs are commonly used to integrate IoT architectures with the respective cloud provider services and backends that further process and store your IoT data.

HiveMQ supports MQTT data integration with the three major cloud providers through flexible and powerful enterprise extensions. Each extension transmits bidirectional MQTT data between IoT devices and cloud providers via the HiveMQ platform. Below is a high-level breakdown:

• The HiveMQ Enterprise Extension for Amazon Kinesis supports seamless bidirectional MQTT data integration with AWS via Amazon Kinesis Data Streams.
• The HiveMQ Enterprise Extension for Google Cloud Pub/Sub supports seamless bidirectional MQTT data integration with Google Cloud via Google Pub/Sub.
• The HiveMQ Enterprise Extension for Kafka allows the full MQTT data integration with Kafka message queues.
• The HiveMQ Enterprise Extension for Kafka also allows full integration with Azures’s Event Hubs service. Learn how to set it up.

  

Adopting the Multi-Cloud for Flexibility and Avoiding Vendor Lock-ins

The multi-cloud strategy ensures your IoT architecture stays flexible and independent of vendors. This empowers your team to make timely decisions if they need to switch cloud providers — enabled by HiveMQ’s enterprise extensions.

You can now integrate your HiveMQ cluster with multiple clouds if your use case requires this setup. A single MQTT message can go to various cloud services, such as AWS and Google Cloud, at the same time.

No matter which cloud provider you choose for your backend processing needs, with the HiveMQ platform, you get an MQTT message broker that is 100% MQTT compatible with all optional features. This ensures your use-case can take full advantage of all available MQTT features. Since our MQTT broker is fully compatible with both the MQTT 3.1.1 and 5.0 specifications, you can plan support for mixed device scenarios without having to think about device migrations. As you purchase and operate newer devices that are MQTT 5.0 conforment you can be assured that they are interoperable with older devices following the MQTT 3.1.1 specification.

Get in Touch with Us to Explore How HiveMQ can Help

Learn more about using HiveMQ’s full-featured MQTT platform to transform MQTT data into Kinesis records for easy bi-directional data integration with AWS and IoT devices. Reach out to us to learn how we can help!

About Nasir Qureshi

Nasir Qureshi is a Senior Product Marketing Manager at HiveMQ. With a passion for working on disruptive technology products, Nasir has helped SaaS companies in their hyper-growth journey for over 3 years now. He holds an MBA from California State University with a major in Technology and Data Management. His interests include IoT devices, networking, data security, and privacy.

Follow Nasir on LinkedIn

Contact Nasir