Enabling Real-Time Environmental Data Monitoring with HiveMQ and Sensor.Community
TL;DR: Why Real-Time Environmental Data Matters
With the HiveMQ Cloud Fully Managed MQTT Broker, we can have open environmental data streamed in real time and available for visualization, downstream analytics, and real-time air quality index (AQI) alerting. We can achieve this functionality with a custom developed containerized API and the Sensor.Community pre-flashed IoT sensor kit. The solution enhances the data collection and reporting of open environmental data. If you ever wanted to have your own end-to-end solution tracking temperature, humidity, pressure and particulate matter concentration, this guide is for you.
Problem: Sensor.Community’s Limited Real-Time Streaming Capabilities
Sensor.Community is a contributor-driven global sensor network that tracks and collects Open Environmental Data. Every enthusiast can install a sensor kit to begin monitoring air data in real-time. The pre-flashed kit sends data to an open-source API. Eventually the data is used for visualizations in a publicly available Grafana instance. If you are a bit more of a data enthusiast, you may wonder: how can you implement real-time monitoring of the incoming data? How can you store the data for historical analysis? The problem is further constrained by the default sensor firmware—it lacks native MQTT integration.
Solution: Extending Sensor.Community with HiveMQ Cloud and a Custom API
The standard IoT device configuration offers the option to send data readings to a custom API:
Therefore, to enable real-time monitoring using a cloud MQTT broker and to add custom data collection, I decided to build a compatible API service. The idea is that at every measurement interval, my API will get the data the IoT sensor kit sends. From there, I can publish the data to an MQTT broker and store it in my own database.
Hardware and Software Requirements for the End-to-End Setup
Hardware:
pre-flashed sensor kit. Here is what a standard kit looks like:
You can buy the kit as a complete set or source each component separately.
Software:
Visual Studio Code
Docker Desktop or similar containerization engine
An account with a cloud provider. I used Azure, but the solution can be adapted for any cloud provider. You will need the following services, or their equivalents:
Container Registry
App Service Plan & Azure Function
Azure SQL Server & Database
A HiveMQ Cloud account for creating a cloud MQTT broker. If you have an existing account, that is fine. If not, you can sign up for the free tier using this link.
Node-RED for visualizing the data. The easiest way is to also run it under Docker.
Hardware Setup: Getting Started with the Sensor.Community Kit
To get started installing and using the kit, follow the guide on the Sensor.Community site. This document will focus on the software side of the setup.
Software Setup: Building the API-to-MQTT Integration Layer
On the software side, the task is to develop a solution that will do the following:
Create a REST API endpoint that can accept requests from the IoT sensor with data payload
Connect to HiveMQ MQTT broker and publish the payload
Connect to and store the data in a database
Wrap all functionality in a container
Run the container in an Azure function app
The full solution code is available at this GitHub repo. In this document, I will only provide a high-level overview of the components of the solution. Here is the overall namespace structure:
└── 📁sensor-community-api-to-mqtt
└── 📁.venv
└── 📁api
└── 📁models
├── __init__.py
├── air_data.py
├── db_model.py
└── 📁routers
├── __init__.py
├── sensor_readings.py
├── __init__.py
├── .env
├── dependencies.py
├── function_app.py
├── main.py
├── publisher.py
├── rds.py
├── sql_client.py
├── transformer.py
├── .gitignore
├── Dockerfile
├── README.md
└── requirements.txtHere is a breakdown of the modules in the solution:
Folder | Module | Function | Explanation |
|---|---|---|---|
/sensor-community-api-to-mqtt | dependencies | Implements HTTP Basic Authentication for the FastAPI application and provides credential verification to protect the endpoint from unauthorized access. | The module secures the API endpoints by requiring clients to provide a username and password with each request. |
/sensor-community-api-to-mqtt | function_app | Configures the FastAPI app to run as an Azure Function with CORS enabled. Creates the necessary Azure Function handler. | Adds CORS middleware to allow cross-origin requests from any domain, wraps the FastAPI app with Mangum for Azure Functions compatibility, and creates an Asynchronous Server Gateway Interface (ASGI) function app that Azure can invoke. |
/sensor-community-api-to-mqtt | main | Creates a FastAPI app with global authentication, includes sensor reading routes, and implements comprehensive logging for requests and validation errors. | Enforces authentication on all endpoints, adds a custom error handler that logs validation failures with raw request bodies for debugging, and logs all incoming requests and responses. |
/sensor-community-api-to-mqtt | publisher | Implements an MQTT client using the HiveMQ Python code template. Connects to a broker over TLS and publishes sensor data payloads to a configured topic. | Establishes a secure connection to an MQTT broker using credentials from environment variables, maintains connection state through callbacks, and publishes JSON-serialized data with QoS 1 for guaranteed delivery while managing connection timeouts or errors. |
/sensor-community-api-to-mqtt | rds | Defines a configuration class for connecting to a SQL Server database using SQLAlchemy with connection parameters loaded from environment variables. | Uses the attrs library to create a validated configuration class that builds a SQLAlchemy connection URL and engine for MS SQL Server with ODBC Driver. |
/sensor-community-api-to-mqtt | sql_client | Provides a SQL database client that stores weather sensor data payloads into a SQL Server database table. | Initializes a SQLAlchemy engine from the relational database service (RDS) configuration, creates the weather_data table if it does not exist, and inserts dictionaries of payload data into the database with automatic connection management and error logging. |
/sensor-community-api-to-mqtt | transformer | Transforms sensor reading objects into standardized dictionary formats by extracting temperature, humidity, and pressure values from sensor data JSON arrays. | Extracts specific sensor values from the sensordatavalues array at fixed indices, packages them with the raw JSON data into a dictionary. Provides placeholder methods for future transformations. |
/sensor-community-api-to-mqtt | Dockerfile | Creates a container image with Python 3.11, installs MS SQL Server ODBC drivers, and configures it to run a FastAPI application. | Uses the Azure Functions Python base image, installs Microsoft ODBC Driver 18 for SQL Server connectivity, installs Python dependencies, sets Azure Functions environment variables, and starts the FastAPI app for both Azure deployment and local testing. |
/.venv | — | Contains the Python virtual environment | — |
/api/models | air_data | Defines Pydantic models for validating incoming sensor data with an ESP8266 chip ID, software version, and a list of sensor measurement values. | Creates a structured data model where SensorReading contains SensorDataValue objects holding various measurements like temperature, pressure, humidity, etc. as key-value pairs. |
/api/models | db_model | Defines a SQLAlchemy table model for storing weather sensor data with temperature, pressure, humidity measurements, and raw JSON data in SQL Server. | Creates a WeatherData table with an auto-incrementing ID, automatic timestamp, decimal columns for the measurements, and a JSON column for the raw sensor data. |
/api/routers | sensor_readings | Defines FastAPI routes for receiving authenticated sensor data via POST, transforming it, and asynchronously publishing to MQTT and storing in SQL Server database. | Creates two endpoints: a root GET endpoint that confirms API status, and a POST endpoint at /sensor-data that validates incoming sensor readings, transforms them to dictionary format, and uses background tasks to connect to HiveMQ, publish the data, store it in the database, then disconnect. |
After building the container and deploying it to the function app, I can see the API service running:
At this point, the API service is ready to accept data payloads from the sensor. From the configuration menu available at a local IP on the network, we must configure the device to send requests to the custom API alongside the default one:
Data Flow: From Sensor to HiveMQ Cloud in Real Time
Having applied the configuration, I can navigate to the HiveMQ Cloud console, where I can see the output from my sensor. All the way from my balcony to the cloud!
Data Visualization
Next, we can use Node-RED or a similar platform supporting MQTT as data source to develop a dashboard displaying the data in real-time. This is a huge benefit enabled by the publish-subscribe nature of the MQTT protocol. I used the handy Node-RED “MQTT in” node to subscribe to the broker in real time. The flow definition is also available in the repo. Here is what the dashboard looks like:
Having everything connected and all software configurations in place, let us take a step back and see what the benefits are of publishing the data to HiveMQ.
Why HiveMQ? Benefits of a Cloud MQTT Broker for Environmental Data Streaming
To create a solid foundation for advanced analytics and monitoring, I chose to publish the data to a cloud-based MQTT broker. While my initial API implementation ran as a Windows service with a locally hosted broker, migrating to a cloud solution offers significant advantages in flexibility and accessibility. A cloud broker provides universal data access from anywhere with internet connectivity, eliminates the need for local infrastructure maintenance, and enables seamless integration with downstream analytics platforms and third-party services.
HiveMQ's cloud offering proved ideal for these requirements—the setup process took less than a minute, and I had a production-ready broker with built-in security, monitoring dashboards, and guaranteed uptime. This cloud-first architecture not only simplifies the current implementation but also creates a foundation for scaling to multiple sensors, implementing distributed data processing, and building collaborative environmental monitoring networks without the constraints of local hardware dependencies.
The usage data is also quite compelling: each payload has a size of approximately 800 bytes, and with the default publishing frequency of 145 seconds, this translates to just 596 publish events per day—roughly half a megabyte of daily traffic. This minimal footprint falls well within the range of HiveMQ's free tier limits, making it a cost-effective solution for individual enthusiasts while maintaining the infrastructure needed for future expansion to multi-sensor networks:
Conclusion: Unlocking New Possibilities for Open Environmental Data
This project successfully demonstrates how a cloud-based MQTT broker can transform the capabilities of Sensor.Community's IoT environmental monitoring system by bridging the gap between limited sensor connectivity firmware and real-time data infrastructure. By developing a containerized FastAPI service that intercepts sensor readings and publishes them to HiveMQ, the solution unlocks critical functionality that wasn't previously available: real-time data visualization through Node-RED dashboards, persistent storage for historical analysis in SQL databases, and a scalable foundation for advanced use cases like machine learning anomaly detection and air quality forecasting.
The architecture can be particularly practical for hobbyists and environmental enthusiasts, as it operates well within HiveMQ's free tier limits while maintaining enterprise-grade security through TLS encryption and authentication. With modest cloud resources and open-source tools, any data developer can build end-to-end IoT monitoring solutions that contribute to global environmental awareness.
Future Enhancements
With the core infrastructure now operational, I am planning several enhancements to further leverage the real-time data capabilities that HiveMQ enables:
Implement dynamic data quality validation that references historical data to identify anomalous data readings
Integrate an LLM agent or a machine learning model for predictive air quality forecasting or anomaly detection
Develop pre-aggregated data products to accelerate downstream analytics and reporting workflows.
Resources
Hristo Hristov
Hristo Hristov is an Industrial Data Engineer specializing in implementing data engineering and analytics solutions for smart manufacturing environments. As an enthusiast technical writer, he explores the practical applications of Azure, HiveMQ, Python, IoT, and AI technologies in industrial settings.
