Apache Kafka is a popular open source streaming platform that makes it easy to share data between enterprise systems and applications. The HiveMQ Enterprise Extension for Kafka makes it possible to send and receive IoT device data with a Kafka cluster.
NEW: HiveMQ Cloud now integrates with Confluent Cloud making it easy to stream IoT data into Confluent Cloud.
With the 4.12 release, the Enterprise Extension for Kafka now has the following additional features:
Added tracing support in the HiveMQ Enterprise Extension for Kafka for MqttToKafkaTransformer customizations that create multiple Kafka records from a single MQTT message.
Added the capability to export information about the HiveMQ Enterprise Extension for Kafka in a diagnostic archive to further facilitate support and
HiveMQ and Kafka Architecture
HiveMQ solves the issues of Kafka for IoT by seamlessly integrating MQTT messages into the Kafka messaging flow. Conversely, Kafka messages can be distributed to HiveMQ and MQTT clients. The HiveMQ broker is extended to include a native implementation of the Kafka protocol. This allows HiveMQ to transpose MQTT messages into the Kafka protocol and vice versa. Our architecture allows for full MQTT support of IoT data plus complete integration with Kafka.
Architectural diagram of HiveMQ and Kafka
Why Is HiveMQ & MQTT Needed for IoT Use Cases
Kafka is well suited for sharing data between enterprise systems and applications located in a data center or in the cloud. However, for IoT use cases there are a number of reasons Kafka is not well suited:
Kafka brokers need to be addressed directly by the client which isn't the case for IoT devices that connect through load balancers.
Kafka clients require a stable IP connection which isn't always the case for IoT devices connecting over unreliable cellular networks.
Kafka does not support large amounts of topics so large IoT deployments with large topics spaces typically don't fit well into Kafka.
Kafka clients are complex and resource intensive so smaller constrained IoT devices often can't run a Kafka client.