Apache Kafka is a popular open source streaming platform that makes it easy to share data between enterprise systems and applications. The HiveMQ Enterprise Extension for Kafka makes it possible to send and receive IoT device data with a Kafka cluster.
HiveMQ and Kafka ArchitectureHiveMQ solves the issues of Kafka for IoT by seamlessly integrating MQTT messages into the Kafka messaging flow. Conversely, Kafka messages can be distributed to HiveMQ and MQTT clients. The HiveMQ broker is extended to include a native implementation of the Kafka protocol. This allows HiveMQ to transpose MQTT messages into the Kafka protocol and vice versa. Our architecture allows for full MQTT support of IoT data plus complete integration with Kafka.
Why Is HiveMQ & MQTT Needed for IoT Use Cases
Kafka is well suited for sharing data between enterprise systems and applications located in a data center or in the cloud. However, for IoT use cases there are a number of reasons Kafka is not well suited:
- Kafka brokers need to be addressed directly by the client which isn't the case for IoT devices that connect through load balancers.
- Kafka clients require a stable IP connection which isn't always the case for IoT devices connecting over unreliable cellular networks.
- Kafka does not support large amounts of topics so large IoT deployments with large topics spaces typically don't fit well into Kafka.
- Kafka clients are complex and resource intensive so smaller constrained IoT devices often can't run a Kafka client.
|Implements the Kafka protocol so MQTT messages can be written to a Kafka topic and conversely Kafka messages can be distributed to multiple MQTT clients.|
|Supports all MQTT 3 and MQTT 5 features, including all QoS levels|
|Select multiple MQTT topic filters per Kafka topic (MQTT topic filters come with full support of MQTT wildcards).|
|Buffer messages at the broker to ensure high-availability and failure tolerance if a Kafka cluster is not available.|
|Monitor MQTT messages that are written to Kafka on the HiveMQ Control Center.|
|Distribute Kafka messages to multiple MQTT clients|
|Map one to many Kafka topics to multiple MQTT topics|
|MQTT PUBLISH packets can be generated dynamically at runtime with variable replacement using parts of the Kafka message|
|Validate messages read from Kafka with the help of a schema registry|
|Use the predefined default configurations or the possibility to set MQTT PUBLISH metadata configuratively|
|... more will follow!|