HiveMQ Kafka Solution

Streaming IoT Data and MQTT Messages to Kafka

Apache Kafka is a popular open source streaming platform that makes it easy to share data between enterprise systems and applications. The HiveMQ Enterprise Extension for Kafka makes it possible to send and receive IoT device data with a Kafka cluster.
Kafka Solution for HiveMQ
NEW: HiveMQ Cloud now integrates with Confluent Cloud making it easy to stream IoT data into Confluent Cloud.
Confluent Integration in HiveMQ Cloud

HiveMQ and Kafka Architecture

HiveMQ solves the issues of Kafka for IoT by seamlessly integrating MQTT messages into the Kafka messaging flow. Conversely, Kafka messages can be distributed to HiveMQ and MQTT clients. The HiveMQ broker is extended to include a native implementation of the Kafka protocol. This allows HiveMQ to transpose MQTT messages into the Kafka protocol and vice versa. Our architecture allows for full MQTT support of IoT data plus complete integration with Kafka.

Architectural diagram of HiveMQ and Kafka

Why Is HiveMQ & MQTT Needed for IoT Use Cases

Kafka is well suited for sharing data between enterprise systems and applications located in a data center or in the cloud. However, for IoT use cases there are a number of reasons Kafka is not well suited:
  • Kafka brokers need to be addressed directly by the client which isn't the case for IoT devices that connect through load balancers.
  • Kafka clients require a stable IP connection which isn't always the case for IoT devices connecting over unreliable cellular networks.
  • Kafka does not support large amounts of topics so large IoT deployments with large topics spaces typically don't fit well into Kafka.
  • Kafka clients are complex and resource intensive so smaller constrained IoT devices often can't run a Kafka client.
Detailed Features
Implements the Kafka protocol so MQTT messages can be written to a Kafka topic and conversely Kafka messages can be distributed to multiple MQTT clients.
Supports all MQTT 3 and MQTT 5 features, including all QoS levels
Select multiple MQTT topic filters per Kafka topic (MQTT topic filters come with full support of MQTT wildcards).
Buffer messages at the broker to ensure high-availability and failure tolerance if a Kafka cluster is not available.
Monitor MQTT messages that are written to Kafka on the HiveMQ Control Center.
Distribute Kafka messages to multiple MQTT clients
Map one to many Kafka topics to multiple MQTT topics
MQTT PUBLISH packets can be generated dynamically at runtime with variable replacement using parts of the Kafka message
Validate messages read from Kafka with the help of a schema registry
Use the predefined default configurations or the possibility to set MQTT PUBLISH metadata configuratively
... more will follow!

Try Now

Download Kafka Extension

Contact us to discuss how HiveMQ can help extend Kafka to IoT device data.