HiveMQ Data Hub Overview
The HiveMQ Data Hub provides mechanisms to define how MQTT data is handled in the HiveMQ broker. This ensures that data quality is assessed at an early stage in the data supply chain, eliminating the need for subscribers to perform resource-intense validation before data reaches downstream devices or upstream services.
HiveMQ Data Hub Capabilities
The HiveMQ Data Hub provides the following capabilities to help you enforce data integrity and quality.
Enforce Behavior Policies
Define behavior policies to manage device interactions with the MQTT broker, including logging, stopping, or transforming actions. Flow-control validates message flow patterns, while the scripting engine enables custom behaviors using JavaScript functions.
Transform Data
Convert or manipulate data formats as it moves through the MQTT broker before reaching consumers. These data transformations enable more processing at the edge, streamlining data integration. For example, convert Fahrenheit to Celsius.
Define Data Policies
Set structured blueprints for data formats and enforce specific rules for incoming messages. Support for JSON and Protobuf formats ensures data integrity, with options to specify requirements, like temperature readings within defined ranges.
Simplify Workflow via Modules
Use specialized Data Hub modules that simplify data transformation and policy enforcement without coding. These modules bundle schemas, scripts, and policies into ready-to-use solutions for real-time automation.
Visualize Data
Utilize the simple user interface to manage schemas, data, and behavioral policies. The dashboard provides an overview of overall quality metrics, making it easy to locate bad actors and bad data sources. Visualize the MQTT data further in tools like Grafana.
Interact with RestAPI
Use HiveMQ’s REST API for programmatic interactions with the HiveMQ Enterprise MQTT broker, enabling seamless integration with other applications. The API supports sending and receiving data as JSON objects.
HiveMQ allows us to securely move business critical data between devices and the cloud at scale. HiveMQ Data Hub now allows us to automatically validate and standardize that data to build a more reliable data pipeline and further grow our business.
Klaas Jan Koopman
/Director Video Back Office & Service Platform, Liberty Global
Data Schema
Create the schema of how data should be formatted. Both JSON and Protobuf formats are currently supported. These can vary from simple to complex. The example on the right is a schema that validates GPS coordinates.
{
"$id": "https://example.com/geographical-location.schema.json",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"title": "Longitude and Latitude Values",
"description": "A geographical coordinate.",
"required": ["latitude", "longitude"],
"type": "object",
"properties": {
"latitude": {
"type": "number",
"minimum": -90,
"maximum": 90
},
"longitude": {
"type": "number",
"minimum": -180,
"maximum": 180
}
}
}Data Policy
An appropriate policy tells HiveMQ how to handle incoming MQTT messages to enforce the rules and structure that the schema outlines. The example on the right drops the message and logs the result, but these can be arbitrarily complex and could re-queue the message. On the re-queue, the system could also apply transformation functions to fix bad data.
{
"id": "com.hivemq.policy.coordinates",
"matching": {
"topicFilter": "coordinates"
},
"validation": {
"validators": [
{
"type": "schema",
"arguments": {
"strategy": "ALL_OF",
"schemas": [
{
"schemaId": "gps_coordinates",
"version": "latest"
}
]
}
}
]
},
"onFailure": {
"pipeline": [
{
"id": "logFailure",
"functionId": "log",
"arguments": {
"level": "WARN",
"message": "${clientId} sent invalid coordinates on topic '${topic}' with result '${validationResult}'"
}
}
]
}
}
Modules
A look inside the HiveMQ Control Center which hosts a library of modules which can help quickly implement functionality like fanning out Sparkplug metrics, or dropping duplicate messages. The modules also helps easily transform data without requiring custom code or complex configuration. This is a growing collection with the potential of customers adding their own modules.
Visualization
A demo use case introduces a quality metric and visualizes it in a Grafana dashboard. An addition to the quality metric is the list of bad clients queried from a PostgreSQL database. A screenshot of the dashboard includes the data quality on the left hand side and list of the top 10 bad clients on the right hand side.
HiveMQ Data Hub Key Benefits
Maximize the business value of data being transported by defining data policies, transformations, and validation requirements to ensure the data is accurate and meets the standards your organization requires.
Faster Business Insights
Provide faster business insights on validated data - stop acting on rogue data and generating more noise than insights.
Increased Data Quality
Ensure data quality standards are centrally defined and enforced across all devices and messages.
Operational Efficiency
Stop bad-acting devices from misusing MQTT connections, sending bad data, and monopolizing resources.
Reduce Costs
Reduce redundant processing and storage costs by only acting on good data. Reduce the impact of acting on bad data.
Improve Data Integrity
Quarantine and further investigate bad data to prevent it from contaminating your systems and ultimately ensuring your data is accurate and reliable.
Data Management
Manage everything in a single system, allowing data to be processed faster and negating the need to manage another standalone system to ensure data quality.
Learn more about HiveMQ Data Hub
Browse the informative HiveMQ Policy Cookbooks repository on GitHub for additional use stories and code samples.
Learn how to measure the quality of your data pipeline including creating MQTT data schemas and policies.
Understand how Data Hub prevents bad actors from potential outages that bring down infrastructures.
Learn how to convert an MQTT Sparkplug message to JSON and fan out metrics to the original topic prefix
Discover how HiveMQ's Sparkplug Module for Data Hub eliminates data bottlenecks and enforces Sparkplug compliance.
