Optimizing Edge-to-Cloud Communications with MQTT
There is an ongoing explosion of network-capable, Internet-connected data and devices. Many electronic devices are Internet-connected now. In industrial and manufacturing spaces, many more unseen devices are used. Smart manufacturing is highly proprietary and, therefore, closed off to Public Internet visibility. In turn, it is harder to estimate the true number of IP-enabled, internet-connected IoT devices in use. The settings are numerous, including factories, energy plants, SCADA systems, and smart, connected cities.
The number of internet and Internet(public) connected IoT devices is exploding. According to an IoT Analytics report, the number of connected IoT devices is estimated to grow to 40 billion by 2030–just around four years from the writing of this article.
The Challenge: Edge-to-Cloud Data Growth at Scale
This vast proliferation of devices is generating a veritable deluge of data. There is an ever-increasing need to efficiently and effectively deliver this data and its insights from the edge to the cloud. Cloud costs are an ever-present reality for many companies. And, cloud costs cannot be ignored. Rather, they must be carefully managed, budgeted, and controlled as they are consumption-based. Devices that transmit data sub-optimally will cost organizations large operational expenses.
Data intelligence is being distributed. In the past, data was centralized. Now, myriad devices are spread across networks. There is a growing need to put data into action where and when it is needed. HiveMQ Pulse is a key enabling technology that manages data intelligence at scale.
MQTT was built specifically for high scalability and efficient communications. MQTT was designed to work over a wide variety of network conditions–slow, intermittent connectivity, and unreliable networks. There are many ways to optimize MQTT messaging. In this article, we examine the motivation and drivers for data growth and the need to optimize effective handling and transmission.
Motivation and Key Drivers
With the deluge of data in modern industry, the goal is not just to collect data. Rather, it is to transform this data into information, contextualize it, and combine this information with domain knowledge. Ultimately, the goal is to transform domain knowledge into insights and wisdom to fuel business growth, usage insights, and better customer and consumer experiences.
⇒ Edge-to-Cloud Data Transmission can get expensive if not done properly!
Therefore, it is important to consider these aspects:
Edge-to-cloud communication challenges - latency, bandwidth, cost, reliability
MQTT is a lightweight messaging protocol for IoT - suited for the task
Optimization for efficient and scalable IoT solutions
Deliver data and resultant insights - more quickly, ever closer to real-time
Control and reduce cloud costs - an ongoing effort
Conserve bandwidth for cost, transmission limits, and device power optimization
Several important and unavoidable factors drive the ongoing need to optimize data in IoT/IIoT and related systems. These include cloud costs, bandwidth, and data voluminosity.
Cloud Costs - Ongoing operational expenses (OpEx) that scale proportionally with data volume and data rate
Bandwidth - the advent and proliferation of 5G, multi-gigabit last-mile networks, has increased bandwidth and transmission.
Data Voluminosity - data is growing at a staggering rate, which challenges the bandwidth constraints. Data growth and bandwidth are in an endless competition.
Key Considerations for Optimized Edge-to-Cloud Communications
Data transmission in IIoT and IoT systems must be reliable, cost-effective, and efficient. This need amplifies as scale increases. For example, 1% inefficiency at 1 Mbps means a waste of 10 Kbps, which may seem acceptable at low scale. The same 1% inefficiency at 100 Mbps translates to a waste of 1 Mbps! Looking at this graphically shows the problem starkly:
Reliability
Ensuring message delivery even in intermittent network conditions (e.g., QoS, persistent sessions, message queuing at the edge).
Handling device and network failures gracefully.
Cloud Cost Optimization
Minimizing data ingress/egress charges.
Reducing compute and storage costs in the cloud through efficient data and message handling.
Impact of optimization techniques on overall cloud expenditure.
Data Deduplication
Strategies to avoid sending redundant data (e.g., report-by-exception, state-based reporting).
Techniques for identifying and discarding duplicate messages at the edge or in the cloud.
Core MQTT Edge-to-Cloud Optimization Techniques
As we investigate edge-to-cloud optimization techniques, it is instructive to look at the MQTT packet structure. As we previously mentioned, MQTT was designed to be lightweight, low-overhead, and performant in constrained devices and networks. We go on to consider payloads, topics, and date filtering,
Understanding MQTT Packet Structure to Minimize Packet Size
As mentioned, MQTT is designed to be low-overhead and lightweight. The MQTT packet is only two (2) bytes at a minimum. Referencing The Essential Guide to MQTT, we see each MQTT packet includes:
Fixed header: 2–5 bytes
Variable header: Topic name, packet identifier, etc of variable size
Payload: Application data of variable size up to the protocol limit of 256 MB
Visually, it is easy to grasp that a minimal MQTT packet is, indeed, very small:
Overhead can become significant if payloads are small and frequent. Remember that overhead is the cost of delivering the data; it does not add additional value and context to the transported payload. Therefore, we should aim to minimize overhead.
MQTT Topics
Recommendations:
Topics should be structured in an organized hierarchy that is human-readable.
Generally, opt toward shorter topic names, but not at the cost of appropriate, unified topic naming.
Do not include redundant context in topic names where it can better live in payloads.
Utilize MQTT5 Topic Aliases–these can reduce payload sizes.
Explanation: Topic aliases reduce header overhead by replacing long topic strings with short, 2-byte numerical identifiers. An MQTT topic can be up to 65,535 bytes long by MQTT specification.
Benefits: Bandwidth savings, especially for frequently published topics. For example, a topic that is 30 bytes long gets reduced to only two (2) bytes resulting in a 15x savings in space in the MQTT packet.
Payload Compression (GZIP, other)
Recommendation: Compress the data payload before sending it over MQTT.
Techniques: Use standard compression algorithms like GZIP or application-specific compression. With payloads like Sparkplug B, typical text JSON gets compressed into Protobuf, a space-efficient binary format. You can use Protobuf for non-Sparkplug payloads, too.
Benefits: Significant reduction in data transfer, lower bandwidth costs.
Consider compression time and compute power vs compress-ratio trade-offs. Edge devices may need to conserve limited power. Edge devices may need to send data rapidly in real-time scenarios, versus striving for maximum compression. Therefore, there is also a time versus bandwidth savings factor to consider.
Report-by-Exception
Concept: Instead of sending data on a fixed schedule, such as every 5 seconds, transmit a message only when the value changes or breaks a threshold. For example, a door sensor only reports its state when it opens or closes. Likewise, a hot/cold sensor only reports data when the thermometer drops below or rises above a set of values, say 32°F / 0°C or 90°F / 32°C. Report by exception can be on either value/range-based or state-based changes.
Effect: This helps limit payload sizes by including only relevant changes in data. Note: the subscribing client may need to hold a memory of the last known value. This can also be augmented or accomplished with a retained message in MQTT on the Broker.
Recommendation: Send data only when a significant change occurs, rather than at fixed intervals.
Techniques:
Define thresholds or deadbands for data changes
Utilize Edge software and features that store current and previous n-values
Benefits: Reduces unnecessary data transmission, conserves bandwidth, and lowers cloud ingestion costs.
Dead-banding
Concept: Dead-banding is an enhancement of report-by-exception and incorporates the notion of thresholds. A message is only sent when a value changes by more than a predefined amount (the "dead band"). A temperature sensor might only report a new value if it changes by more than 0.5°C from the last reported value, filtering out minor, insignificant fluctuations. The following diagram illustrates deadband.
Visualization of Deadband
Explanation: Dead-banding can be considered similar to filtering in both time and data value domains. It is a specific form of report-by-exception where data is only sent if it moves outside a predefined "deadband" or range around the last reported value. A deadband is a threshold that prevents publishers from sending updates if a value hasn't changed significantly from the last published value.
Benefits: This further refines data transmission, ideal for sensor readings with minor fluctuations. Noise in the system is reduced and data quality can be improved in turn.
Effect: Dead-banding can greatly increase data quality and reduce data transmission rates by filtering noise and insignificant value changes. In this way, the data that are transmitted are data that are truly of interest.
Recommendation: Implement dead-band filters on data that is high resolution or may change often, where you care about significant changes in data values over time
Example: Implementing Deadband Filtering on HiveMQ Edge Using Data Hub
The following is an example of a HiveMQ Data Hub policy and script that implements deadband filtering that detects and filters samples greater than two (2) standard deviations of the mean. It uses an algorithm called Welford’s Algorithm that is computationally efficient. It calculates the variance as data is collected, in one pass, without the need to store large amounts of samples.
Again, there is often a tradeoff between edge compute capacity, power draw, and transmission factors that need to be considered. Welford’s algorithm is useful at the edge since it is computationally efficient and delivers a useful filtering function. There are many other techniques and algorithms that can be used to assess and compare measured signals against deadband.
Note that these example scripts work on HiveMQ 4.38 or newer due to the dependence on the Client Connection state. These scripts should work on HiveMQ Edge, too.
Schema:
{
"id": "deadband-policy-for-sensors",
"matching": {
"topicFilter": "sensors/+/temperature"
},
"validation": {
"validators": [
{
"type": "schema",
"arguments": {
"schemaId": "sensor-data-schema",
"strategy": "ALL"
}
},
{
"type": "script",
"arguments": {
"script": "ZnVuY3 <<abbreviated for space >> wp9"
}
}
]
},
"onSuccess": {
"pipeline": [
{
"id": "update-state-and-forward",
"type": "script",
"arguments": {
"script": "ZnVu…. <<abbreviated for space >> =="
}
}
]
}
}Data Hub Script:
function validate(publish) {
// Require at least 10 samples from a client before the filter becomes active.
const MIN_SAMPLES = 10;
// The number of standard deviations from the mean to consider a value valid.
const STD_DEV_FACTOR = 2.0;
// Get the unique client ID from the context of the MQTT PUBLISH packet.
const clientId = publish.context.clientId;
// Create unique state keys for this specific client to ensure its stats are isolated.
const countKey = `${clientId}-count`;
const meanKey = `${clientId}-mean`;
const m2Key = `${clientId}-m2`;
// Get statistics for this specific client from the policy's state.
// If a key doesn't exist (e.g., for a new client), default to 0.
const count = state.get(countKey) || 0;
const mean = state.get(meanKey) || 0;
const m2 = state.get(m2Key) || 0;
// If this specific client hasn't sent enough samples yet, always accept the value
// to help build its initial statistical model.
if (count < MIN_SAMPLES) {
return true;
}
const currentValue = publish.payload.value;
// Calculate the current variance and standard deviation for this client.
const variance = m2 / count;
const stdDev = Math.sqrt(variance);
// Define the upper and lower bounds of the acceptable range.
const lowerBound = mean - (STD_DEV_FACTOR * stdDev);
const upperBound = mean + (STD_DEV_FACTOR * stdDev);
// The validation passes if the current value is within the acceptable range.
return currentValue >= lowerBound && currentValue <= upperBound;
}OnSuccess Script:
function transform(publish, context) {
// Get the unique client ID from the context.
const clientId = publish.context.clientId;
// Create unique state keys for this specific client.
const countKey = `${clientId}-count`;
const meanKey = `${clientId}-mean`;
const m2Key = `${clientId}-m2`;
// Retrieve current statistics for this client from state.
let count = state.get(countKey) || 0;
let mean = state.get(meanKey) || 0;
let m2 = state.get(m2Key) || 0;
const currentValue = publish.payload.value;
// Update the running statistics using Welford's online algorithm.
// This allows us to calculate the new mean and variance without storing all previous values.
count++;
const delta = currentValue - mean;
mean += delta / count;
const delta2 = currentValue - mean;
m2 += delta * delta2;
// Store the updated statistics back into the policy's state using the client-specific keys.
state.put(countKey, count);
state.put(meanKey, mean);
state.put(m2Key, m2);
// Return the original data to allow the message to be published.
return publish;
}MQTT Quality-of-Service (QoS) Levels
It is important to choose the appropriate Quality of Service (QoS) level(s) for your MQTT-capable devices and systems. Quality of Service levels help by balancing reliability with network overhead. This selection has a direct resource impact that shows in bandwidth utilization, energy consumption, and cloud costs. See the analysis section for details.
QoS 0 ("Fire and Forget"):
Lowest overhead, but no guarantee of delivery. Best for non-critical, repetitive sensor data. Here data can be transmitted repeatedly with no explicit semantics for knowing how many times data has been sent, if it has been received, processed, and more.
QoS 0 has the smallest overhead, no ACKs. Best for high-frequency, low-criticality data.
QoS 1 ("At Least Once"): Guarantees the message will be delivered, but duplicates are possible. This is the most common level. It forms a balance point between reliability and performance. Receivers of data, subscribers in MQTT parlance, must be able to handle duplicates. Here though, delivery is guaranteed, preventing the possible “spamming” condition with QoS 1.
QoS 2 ("Exactly Once"): Highest reliability, ensuring the message is delivered exactly once. It requires a more complex four-part handshake, increasing overhead. Use it only for critical commands or transactions where duplicates are unacceptable.
QoS 1/2: This has extra bytes for packet identifiers and ACKs. Use these service levels only when delivery guarantees matter. Overhead and turns in communication can add up quickly at scale, greatly impacting overall bandwidth consumption.
If you are looking to get clarity on how MQTT QoS truly operates, read our blog Debunking Common MQTT QoS Misconceptions.
Batching and Bundling MQTT Packets and Messages
Concept: Combine multiple sensor readings or data points from an edge device into a single, structured MQTT message, such as a JSON object or a protobuf buffer. This is highly efficient because it reduces the overhead associated with individual TCP/IP and MQTT headers for each small piece of data.
Effect: It is more efficient to bundle a group of smaller messages into one larger transmission. Aggregating multiple small messages into a single larger message before publishing.
Benefits: Batching reduces per-message overhead, improves network efficiency, conserves energy at the edge where power may be limited, and lowers connection and processing costs on the cloud side.
Recommendation: Group data points. Hold this data until a significant amount has accumulated or a time limit has been reached.
Other Optimization Techniques and Principles
Here are few other optimization techniques and principles for edge-to-cloud communication with MQTT:
Session Management:
Explanation: Using persistent sessions (clean session = false) to maintain subscriptions and unacknowledged messages across disconnections.
Benefits: Improved reliability and message delivery guarantee.
Efficient Data Serialization:
Explanation: Using compact data formats (e.g., Protocol Buffers, MessagePack) instead of verbose ones (e.g., JSON, XML) for payloads.
Benefits: Reduces payload size, faster parsing.
Edge Processing/Filtering:
Explanation: Performing data aggregation, filtering, and analysis directly at the edge before sending to the cloud.
Benefits: Reduces data volume sent to the cloud, minimizes cloud processing, and lowers costs.
Measurement and Testing
Before you begin optimizing your edge systems, it is important to measure baseline factors such as cost, data transmission rate, average payload size, and cloud utilization and costs. This will give you a number to compare when you start making changes.
As you work to implement the optimization techniques, it is important to measure the effect of these changes on the edge systems, network bandwidth, and cloud cost and utilization consoles.
Consider using a tool like HiveMQ Swarm to manage your load testing scenarios. With Swarm, you can create scenarios and conduct simulation tests of your changes before you deploy them to the field. Even a test environment with edge-to-cloud connectivity can drive cloud costs. Swarm will let you validate your test environments and scenarios in an isolated environment before proceeding to large scale testing. This action can save you lots of time, headaches, and money.
Analysis: Edge-to-Cloud Communication
In this section, we look at some of the effects of edge-to-cloud optimization parameters. The example analyses are shown in terms of network performance, application chattiness, energy consumption, and cloud cost analysis. Note that these are primarily representative to demonstrate the need for optimization derived from analytic models. The cost and energy figures should not be taken as actuals that would apply necessarily to your environments.
Overhead-to-Payload, “Chattiness”, and Energy Consumption at Edge of QoS Levels
Generally, consider mixing QoS levels. Use QoS 0 for less critical data, data that is repeated, or does not change often or change does not need to be dealt with frequently–atmospheric temperature for a weather station–reporting at 5-minute intervals at QoS 0 may be appropriate. QoS 1/2 may provide no additional benefit.
However, temperature control for an oven or solder bath is critical to the process. So, at least QoS 1 should be selected. QoS 0 can often be used for bulk telemetry with QoS 0. You can use QoS 1 for alerts related to telemetry that require decisive action and must be delivered promptly. QoS 2 data should be reserved for critical data that must be delivered reliably and exactly once. This is your most expensive transmission option, so save it for where it really matters.
The following chart shows MQTT overhead-to-payload ratios by QoS level. With the high peak on the left, we can see that overhead is very high at small payloads. When payloads exceed 300 bytes, the impact of the overhead diminishes significantly. Still, QoS2 carries a sustained overhead that is significantly higher than QoS 0,1. These overheads add up as traffic scales up and cannot be ignored.
Energy Consumption
We can clearly see that there is a dramatic difference in energy consumption measured in Watt-hours (Wh) for QoS 0,1,2 over the period of 1 month when considering constrained devices at the edge at a per-device view:
QoS 2 is nearly 300% more costly in terms of energy than QoS 0!
Using October 2025 energy cost in US Dollars, consumer estimates of kWh pricing, we can see a drastic increase in edge energy costs for 1,000 low-power devices:
| QoS Level | Energy Consumption (Wh) per Device | Total Energy (kWh) for 1,000 Devices | Estimated Cost per Month | Estimated Cost per Month for 1,000 Devices | Estimated Cost per Month for 1,000 Devices % Difference (vs. QoS 0) |
|---|---|---|---|---|---|
| QoS 0 | 182 | 181,800 | $31.7 | $31,760.00 | - |
| QoS 1 | 360 | 360,000 | $62.90 | $62,900.00 | 98.02% |
| QoS 2 | 715 | 715,440 | $124.99 | $124,990.00 | 293.42% |
Cloud Cost Analysis
The following example shows a representative model of cloud costs for a dense, high message rate scenario of 1000 devices sending 10 messages per second.
The following chart shows the same representative scenario using list pricing for HiveMQ Cloud.
Conclusion
The increasing proliferation of Internet-connected devices has led to an explosion of data, making the optimization of edge-to-cloud communications essential. Efficient data transmission is central in managing cloud costs, conserving bandwidth, and transforming raw data into valuable insights. These insights drive business growth, improve system quality, and improve customer experiences.
Several key MQTT and general optimization strategies can be employed to achieve these objectives. Techniques include using MQTT5 topic aliases, payload compression like GZIP or Protobuf, and smart data transmission techniques such as report-by-exception and dead-banding, which reduce unnecessary data transfer. Batching and bundling multiple data points into a single message also significantly decreases overhead.
Selecting Quality of Service (QoS) levels based on data criticality, implementing robust session management, and utilizing efficient data serialization formats like protobuf encoding are crucial. Edge processing and filtering also play a vital role in reducing the volume of data sent to the cloud, thereby minimizing processing, transmission, and storage costs.
Looking ahead, the trends in edge computing and IoT communication will continue to emphasize intelligent data handling at the source, further refining these optimization techniques. Even with advances in communication networks, the primary goal remains the same–to achieve a balance between reliability, cost-effectiveness, and efficiency in a rapidly expanding connected world. Reach out to us if you’d like to optimize your edge-to-cloud communication.
Resources
Bill Sommers
Bill Sommers is a Technical Account Manager at HiveMQ, where he champions customer success by bridging technical expertise with IoT innovation. With a strong background in capacity planning, Kubernetes, cloud-native integration, and microservices, Bill brings extensive experience across diverse domains, including healthcare, financial services, academia, and the public sector. At HiveMQ, he guides customers in leveraging MQTT, HiveMQ, UNS, and Sparkplug to drive digital transformation and Industry 4.0 initiatives. A skilled advocate for customer needs, he ensures seamless technical support, fosters satisfaction, and contributes to the MQTT community through technical insights and code contributions.
