Extending Confluent Cloud to Ingest Data from Millions of IoT Devices
Confluent Cloud is the popular fully-managed, cloud-native event streaming platform powered by Confluent. It makes it easy to deploy and manage Apache Kafka clusters in the cloud and stream data between different enterprise services. Many companies depend on Apache Kafka as their central messaging and real-time streaming platform, enjoying the freedom to concentrate on their use case while they let Confluent run and operate the service.
Extending Confluent Cloud to IoT
As a real-time streaming platform, Confluent Cloud has been optimized to stream data as fast as possible between enterprise services in a scalable manner. However, ingesting data from hundreds of thousands or even millions of IoT devices into Confluent Cloud comes with additional challenges where the Kafka architecture is not suitable.
Concurrently connecting tens of thousands or even millions of devices to a Confluent Cloud Kafka cluster is very unusual and often not possible. Kafka clients need a stable TCP connection to the Kafka cluster. In most IoT use cases, such a connection is not possible due to unreliable networks and connections that need to be constantly re-established.
The Kafka protocol also lacks some key IoT features such as keep-alive and last will and testament. Additionally, Kafka does not support the large number of topics that are typical in many IoT use cases.
A detailed list and explanation of these challenges can be found in our previous article about Streaming IoT data and MQTT messages. In the end, the most scalable, reliable, and secure way to ingest data from IoT devices into your Confluent Cloud Kafka cluster is to use a mature, enterprise-grade MQTT broker, that can be extended to act as a first-class Kafka client.
HiveMQ Cloud Streams IoT Data to Confluent Cloud
HiveMQ Cloud, which was introduced early this year, is a completely managed MQTT broker in the cloud. The vision has always been for HiveMQ Cloud to be a cloud native messaging platform that integrates with a variety of cloud services. Now, we are thrilled to announce that HiveMQ Cloud integrates seamlessly with Confluent Cloud. The Confluent Cloud Integration for HiveMQ Cloud provides an easy and simple way to ingest IoT device data into your Confluent Cloud Kafka cluster.
MQTT message ingestion
All IoT data that is published to HiveMQ can be converted to native Kafka messages and forwarded to your Kafka cluster running on Confluent Cloud. This makes it easy to process data from your IoT devices into backend services.
Full observability
The Confluent Cloud Integration adds an additional page to the HiveMQ Control Center that enables full observability of the message streaming. This additional view provides a transparent overview of incoming MQTT messages and transformed and forwarded Kafka messages.
End-to-end message delivery guarantees
HiveMQ Cloud’s Confluent Cloud Integration provides end-to-end message delivery guarantees. It will queue messages if the network connection to Confluent Cloud is temporarily unavailable so that messages are not lost.
Flexible topic mappings
IoT device data can include millions of MQTT message topics. Our integration with Confluent Cloud makes it easy to map MQTT message topics to a smaller number of Kafka message topics. All you need to do is use MQTT wildcards to filter and route MQTT message data to the corresponding Kafka topic. We support mapping MQTT message topics to one or more Kafka message topics.
Easy and simple configuration
Setting up and configuring services can be tough and time-consuming. That’s why we built the Confluent Cloud integration with an easy and simple configuration that lets you start and use your integration in minutes.
Next Step
To get started sign up for a free trial of HiveMQ Cloud now. The Confluent Cloud integration is included in your 7 day free trial of HiveMQ Cloud. You should already have an Confluent Cloud account with a running Kafka cluster.
Integrating HiveMQ Cloud with your Confluent Cloud Kafka cluster takes just a few steps:
- Create a Starter Package in your HiveMQ Cloud Portal
- Once cluster creation is complete, select Integrations in the cluster details view
- Add the Confluent Cloud Integration to your running HiveMQ Cloud Starter package
- Configure the integration accordingly
- Connect IoT devices to HiveMQ Cloud and start sending messages
Take a look at our step-by-step getting started guide and start to ingest your IoT data into your Confluent Cloud Kafka cluster today.
About Magi Erber
Magi Erber is Senior Product Manager at HiveMQ. She loves creating software that delights customers and helps them realizing innovative IoT solutions.
Contact Magi
