HiveMQ Data Hub Overview
The HiveMQ Data Hub provides mechanisms to define how MQTT data is handled in the HiveMQ broker. This ensures that data quality is assessed at an early stage in the data supply chain, eliminating the need for subscribers to perform resource-intense validation before data reaches downstream devices or upstream services.
HiveMQ Data Hub Capabilities
The HiveMQ Data Hub provides the following capabilities to help you enforce data integrity and quality.
Enforce Behavior Policies
Define a policy that dictates agreed upon behaviors for how devices should work with the MQTT broker by logging, stopping, or transforming the behaviors. Flow-control validates in-flight message flow patterns, and the scripting engine allows you to implement behaviors tailored to your specific needs via JavaScript functions.
Transform Data
Convert or manipulate raw or structured data into the desired format while it moves through the MQTT broker and before it reaches consumers. Transformations move more operations to the edge so data can be standardized. For example, convert Fahrenheit to Celsius, or something more sophisticated such as data manipulation using JavaScript.
Visualize Data
Utilize the simple user interface to manage schemas, data, and behavioral policies. The dashboard provides an overview of overall quality metrics, making it easy to locate bad actors and bad data sources. Visualize the MQTT data further in tools like Grafana.
Data Schemas
Create the blueprint for how data is formatted. Both JSON and Protobuf formats are currently supported.
Data Policies
Define the set of rules and guidelines to enforce how data and messages should be expected, like requiring a temperature between 100-1200.
RestAPI
The HiveMQ REST API provides an interface for applications to interact programmatically with the HiveMQ Enterprise MQTT broker.
HiveMQ allows us to securely move business critical data between devices and the cloud at scale. HiveMQ Data Hub now allows us to automatically validate and standardize that data to build a more reliable data pipeline and further grow our business.
Klaas Jan Koopman
/Director Video Back Office & Service Platform, Liberty Global
Data Schema
Create the schema of how data should be formatted. Both JSON and Protobuf formats are currently supported. These can vary from simple to complex. The example on the right is a schema that validates GPS coordinates.
{
"$id": "https://example.com/geographical-location.schema.json",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"title": "Longitude and Latitude Values",
"description": "A geographical coordinate.",
"required": ["latitude", "longitude"],
"type": "object",
"properties": {
"latitude": {
"type": "number",
"minimum": -90,
"maximum": 90
},
"longitude": {
"type": "number",
"minimum": -180,
"maximum": 180
}
}
}Data Policy
An appropriate policy tells HiveMQ how to handle incoming MQTT messages to enforce the rules and structure that the schema outlines. The example on the right drops the message and logs the result, but these can be arbitrarily complex and could re-queue the message. On the re-queue, the system could also apply transformation functions to fix bad data.
{
"id": "com.hivemq.policy.coordinates",
"matching": {
"topicFilter": "coordinates"
},
"validation": {
"validators": [
{
"type": "schema",
"arguments": {
"strategy": "ALL_OF",
"schemas": [
{
"schemaId": "gps_coordinates",
"version": "latest"
}
]
}
}
]
},
"onFailure": {
"pipeline": [
{
"id": "logFailure",
"functionId": "log",
"arguments": {
"level": "WARN",
"message": "${clientId} sent invalid coordinates on topic '${topic}' with result '${validationResult}'"
}
}
]
}
}
Visualization
A demo use case introduces a quality metric and visualizes it in a Grafana dashboard. An addition to the quality metric is the list of bad clients queried from a PostgreSQL database. A screenshot of the dashboard includes the data quality on the left hand side and list of the top 10 bad clients on the right hand side.
HiveMQ Data Hub Key Benefits
Maximize the business value of data being transported by defining data policies, transformations, and validation requirements to ensure the data is accurate and meets the standards your organization requires.
Faster Business Insights
Provide faster business insights on validated data - stop acting on rogue data and generating more noise than insights.
Increased Data Quality
Ensure data quality standards are centrally defined and enforced across all devices and messages.
Operational Efficiency
Stop bad-acting devices from misusing MQTT connections, sending bad data, and monopolizing resources.
Reduce Costs
Reduce redundant processing and storage costs by only acting on good data. Reduce the impact of acting on bad data.
Improve Data Integrity
Quarantine and further investigate bad data to prevent it from contaminating your systems and ultimately ensuring your data is accurate and reliable.
Data Management
Manage everything in a single system, allowing data to be processed faster and negating the need to manage another standalone system to ensure data quality.
Learn more about HiveMQ Data Hub
Browse the informative HiveMQ Policy Cookbooks repository on GitHub for additional use stories and code samples.
Learn how to measure the quality of your data pipeline including creating MQTT data schemas and policies.
Understand how Data Hub prevents bad actors from potential outages that bring down infrastructures.
