Skip to content

Stream IoT Data Between MQTT and Kafka with HiveMQ Cloud

by Shashank Sharma
7 min read

MQTT and Apache Kafka are often used together to enhance the functionality of IoT and Machine-to-Machine communications. You commonly see them paired in the following use cases:

  • Data collection: MQTT is used to collect data from IoT devices and publish it to a Kafka broker, where it is processed, analyzed, and stored for future use.

  • Real-time processing: Using MQTT and Kafka, organizations build real-time data processing pipelines that handle large amounts of incoming data from IoT devices.

With HiveMQ Cloud Starter, you can now stream data to Apache Kafka with our new Kafka integration. 

The HiveMQ Cloud Kafka Integration

There are five (5) easy steps to ingest data from your IoT devices with the Apache Kafka service of your choice. These can be broadly divided into:

  • Connection configuration parameters

  • Topic mapping parameters 

The connection configuration parameters help establish a secure connection between HiveMQ Cloud and your Apache Kafka cluster. Topic mappings let you set up the bidirectional data flow between your MQTT cluster and Apache Kafka.

But first, you must find the Kafka extension in the “Integrations” tab inside your HiveMQ Cloud cluster. This integration is available with HiveMQ Cloud Starter. 

Integrations tab in HiveMQ CloudIntegrations tab in HiveMQ Cloud

Configuring Your Kafka Integration

Now you are ready to dive into the five steps:

1. Connect HiveMQ Cloud with the Kafka service of your choice: To connect, you need a list of bootstrap servers for your Kafka cluster so the integration can fetch the initial metadata about your Kafka cluster.

Connect HiveMQ Cloud with the Kafka service of your choice

2. Secure the connection: Now you need to add your Kafka credentials. This helps ensure there is a secure connection between HiveMQ Cloud and Kafka.

Secure the connection

3. Send data from HiveMQ to Kafka: Once you set up and secure the connection, you can choose what data to forward from your IoT devices. This requires mapping topics from HiveMQ Cloud to your Kafka cluster.

The source topic is the MQTT topic you want to send from your HiveMQ cluster. The destination topics are the Kafka topic receiving the messages that your HiveMQ cluster sent. 

Send data from HiveMQ to Kafka

4. Establish bidirectional communication: For bidirectional communication between Kafka and HiveMQ, you can configure the Kafka cluster to HiveMQ Cloud similarly as you define the topic mapping from HiveMQ Cloud to your Kafka cluster. You can also use ${KAFKA_TOPIC}, ${KAFKA_KEY}, or ${KAFKA_VALUE} as variable part in the MQTT topic. This helps you to efficiently map one Kafka topic to one or more MQTT topics.

In this case, the source topic represents the Kafka topic from which the integration should read messages. These messages are then published with the defined destination topic on your HiveMQ Cloud MQTT broker cluster.

Establish bidirectional communication

5. Enable the configuration: You can start the data flow between the HiveMQ Cloud cluster and your Kafka cluster by selecting the “enable” button. Once active, you see a green banner at the top, indicating that your integration is active.

Enable the configurationIf you've followed these five steps, you should now be able to employ Apache Kafka with HiveMQ Cloud to use data from your IoT devices for bidirectional communication. 

Watch the interactive demo

Validating the Data Flow With the Web Client

Now that you have enabled your integration, you can use the inbuilt web client to quickly validate the data flow between HiveMQ Cloud and Kafka server. 

For that, you need to enter your credentials and add the same topic name under “Publish message” section as the one entered in the configuration. Then, you can add test message payload and click the Publish button as shown in the image below. 

Validating the Data Flow with the Web ClientOnce you see the message being published, you can go to your Kafka server and validate that the data has been flown into the mapped Kafka Topic. For this blog post, we used Confluent cloud but you can use any Kafka server of your choice.  Kafka Server for validate the streaming of IoT data

We’ve captured this information for you in a 2 minute video as well.

Get Started 

To access this Kafka-HiveMQ Cloud functionality, all you need to do is sign up for a 15-day free trial with HiveMQ Cloud Starter.

Sign Up for HiveMQ Cloud

Shashank Sharma

Shashank Sharma is a product marketing manager at HiveMQ. He is passionate about technology, supporting customers, and enabling developer-centric workflows. He focuses on the HiveMQ Cloud offerings and has previous experience in application software tooling, autonomous driving, and numerical computing.

  • Contact Shashank Sharma via e-mail
HiveMQ logo
Review HiveMQ on G2