HiveMQ Kafka Solution

Apache Kafka is a popular open source streaming platform that makes it easy to share data between enterprise systems and applications. The HiveMQ Enterprise Extension for Kafka makes it possible to send and receive IoT device data with a Kafka cluster.

NEW: HiveMQ Cloud now integrates with Confluent Cloud making it easy to stream IoT data into Confluent Cloud.
With the 4.12 release, the Enterprise Extension for Kafka now has the following additional features:

Added tracing support in the HiveMQ Enterprise Extension for Kafka for MqttToKafkaTransformer customizations that create multiple Kafka records from a single MQTT message.
Added the capability to export information about the HiveMQ Enterprise Extension for Kafka in a diagnostic archive to further facilitate support and troubleshooting.

HiveMQ and Kafka Architecture

HiveMQ solves the issues of Kafka for IoT by seamlessly integrating MQTT messages into the Kafka messaging flow. Conversely, Kafka messages can be distributed to HiveMQ and MQTT clients. The HiveMQ broker is extended to include a native implementation of the Kafka protocol. This allows HiveMQ to transpose MQTT messages into the Kafka protocol and vice versa. Our architecture allows for full MQTT support of IoT data plus complete integration with Kafka.

Architectural diagram of HiveMQ and Kafka

Why Is HiveMQ & MQTT Needed for IoT Use Cases

Kafka is well suited for sharing data between enterprise systems and applications located in a data center or in the cloud. However, for IoT use cases there are a number of reasons Kafka is not well suited:

Kafka brokers need to be addressed directly by the client which isn't the case for IoT devices that connect through load balancers.
Kafka clients require a stable IP connection which isn't always the case for IoT devices connecting over unreliable cellular networks.
Kafka does not support large amounts of topics so large IoT deployments with large topics spaces typically don't fit well into Kafka.
Kafka clients are complex and resource intensive so smaller constrained IoT devices often can't run a Kafka client.

More Resources

Blog Article "Streaming IoT Data and MQTT Messages"

Documentation

Download HiveMQ Enterprise Extension for Apache Kafka ®

Webinar with Confluent: Best Practices for Streaming IoT Data with MQTT and Apache Kafka ®

Video: HiveMQ and Apache Kafka: How to integrate both worlds

Detailed Features
	Implements the Kafka protocol so MQTT messages can be written to a Kafka topic and conversely Kafka messages can be distributed to multiple MQTT clients.
	Supports all MQTT 3 and MQTT 5 features, including all QoS levels
	Select multiple MQTT topic filters per Kafka topic (MQTT topic filters come with full support of MQTT wildcards).
	Buffer messages at the broker to ensure high-availability and failure tolerance if a Kafka cluster is not available.
	Monitor MQTT messages that are written to Kafka on the HiveMQ Control Center.
	Distribute Kafka messages to multiple MQTT clients
	Map one to many Kafka topics to multiple MQTT topics.
	Programmatic Customization API to specify MQTT to Kafka topic mapping and custom message transformation.
	MQTT PUBLISH packets can be generated dynamically at runtime with variable replacement using parts of the Kafka message
	Validate messages read from Kafka with the help of a schema registry
	Use the predefined default configurations or the possibility to set MQTT PUBLISH metadata configuratively
	Added tracing support in the extension for MqttToKafkaTransformer customizations; that can create multiple Kafka records from a single MQTT message.
	Export information about the extension in a diagnostic archive to further facilitate support and troubleshooting.
... more will follow!

Try Now

Try HiveMQ Extension for Kafka

Download the Kafka Extension for HiveMQ