MQTT on Raspberry Pi: Send Sensor Data to HiveMQ Cloud with Java and Pi4J
Welcome to the three-part blog series MQTT on Raspberry Pi using HiveMQ Cloud. These blog posts are written by Frank Delporte, the author of a popular book Getting Started with Java on the Raspberry Pi. In this post, Frank explains why he chose HiveMQ Cloud, the cloud-hosted MQTT broker, to send sensor data from Raspberry Pi to it using Java and Pi4J.
Introduction: Why Use HiveMQ Cloud MQTT Broker?
A few years ago, I did my first experiments with an MQTT server (Mosquitto) running on a Raspberry Pi to connect an Arduino and Raspberry Pi for my son’s drum booth. In this experiment, I created a touchscreen controller to control relays and LED strips and different lights of the drum booth. My book describes the complete process.
In this series of posts, I am going to take a different approach with an online MQTT-compatible service, HiveMQ Cloud. The biggest advantage of HiveMQ Cloud: There’s no need for an always-on server that one must self-manage. Yes, Mosquitto runs on an inexpensive Raspberry Pi. Still, we require it to keep it up-and-running if we use it as the “central hub” to distribute data.
Another big advantage of HiveMQ Cloud: It is free to connect up to 100 devices! Even for the most enthusiastic maker, that’s a lot of microcontrollers or computers! You can find example code for many programming languages from HiveMQ. In this blog series, I will focus on a few examples with Raspberry Pi and Java.
Setting Up a HiveMQ Cloud Account
Let’s start with a free HiveMQ Cloud account.
Go to HiveMQ Cloud Sign Up, click on Sign Up Now and create a login.
Once logged in, you will be presented with your HiveMQ MQTT cluster.
Click on the Manage Cluster button. You will see the details of your HiveMQ Cloud instance.
Go to the Access Management section and create a username and password for the credentials, which we will use in our Java application.
About Raspberry Pi
The Raspberry Pi isn’t just a small, inexpensive computer-for-everyone. The latest version, Rasperry Pi 4, has evolved to a powerful machine. This version can replace a full-size desktop or notebook for many use-cases.
The initial goal of the Raspberry Pi project was to build an inexpensive PC that is affordable for all. The first version could be connected directly to a television to provide a computer to those who couldn’t afford a monitor. Now different versions of the Raspberry Pi are available, depending on your budget, the connections you need, the processor, the amount of memory, etc.
In 2019, the Raspberry Pi 4 was released with a 1.5 GHz 64-bit quad-core ARM processor. This board is available with 1, 2, 4 or 8GB of memory. The prices range between 25 to 90€.
In this series, I will be using a Raspberry Pi 4. You can do exactly the same with the other types of board. The only thing you need to take care of when using a Raspberry Pi Zero of the first generation (a new one was released in November 2021) or older full-size boards, is to use an ARMv6 compatible Java JDK or runtime. More info is available on How to install and use Java 11 and JavaFX 11 on Raspberry Pi boards with ARMv6 processor.
The “magic part” of the Raspberry Pi is the 40-pins header, which allows you to connect a wide range of electronic components, like LEDs, buttons, sensors, servos, relays, screens, etc. That’s exactly what I want to use in this post to send sensor data to HiveMQ Cloud.
Java on Raspberry Pi
When you start with a new Raspberry Pi, you will need to “burn” an Operating System to an SD card. On the software page of the Raspberry Pi, you can find the “Imager” tool. With this tool, you can burn one of the pre-defined OS versions, or select a downloaded image from another site. If you select “Raspberry Pi OS (other)” and “Raspberry Pi OS Full (32-bit)”, you will have a full Linux desktop environment with extra programming tools.
What’s even more important for us is that Java is pre-installed! When you open the terminal and check the version with
java -versions, you will get the below result (depending on the build of Raspberry Pi OS):
The above terminal information indicates that we are fully prepared to develop and run Java applications on our Raspberry Pi!
TIP: All three videos accompanying this MQTT on Raspberry Pi blog post series are avialable on our YouTube Channel. Be sure to watch and complement your knowledge.
The Pi4J Project
The Pi4J project (started in 2012) aims to unite Java programming with electronics. By using the Pi4J dependency in a project, electronic components connected to the GPIO-pins (General Purpose Input/Output) of the Raspberry Pi can be controlled as objects in the Java-code. Pi4J uses native libraries to control the GPIOs so you - as a programmer - donʼt need to be fully aware of all the “magic” that relates to hardware communication.
The History of Pi4J Project
In early 2021, two new versions of Pi4J were released:
- Version 1.3: Released in January 2021, it added support for the newest Raspberry Pi boards (4, 400, and Compute 4)
- Version 1.4: Released in March 2021, it targets Java 11 and removed support for other boards and components
The full history of V1 is available on the Pi4J website. Since 2019, the work on a complete new version 2 of Pi4J is ongoing to bring modules and improve the architecture. This has made the project easier to maintain, test, and release. In August 2021, the first release of this new version was made. That’s the version I am using for this Java project.
Together with V2, a new documentation was published where you can find a lot of information about electronics and how to use Java and JavaFX on the Raspberry Pi.
The Java Project: Using HiveMQ Cloud and CrowPi
The first application we will create is a data-publisher and the full sources are available on GitHub. We will read the data from different sensors and send their values to HiveMQ Cloud. Because every “maker” will have his/her own ideas and wishes for a project, we will not focus on components and wiring, but use an easy starter kit that is perfect for this demo project: the CrowPi. This pre-wired kit makes it very easy to get started with electronics programming as all the components are pre-assembled and wired. At least that’s one thing you don’t need to worry about or can mess up.😉
The CrowPi Project by Swiss FHNW University
Students from the Swiss FHNW University created a full Java-project for the CrowPi to demonstrate how all the components in this kit can be controlled with Java. Part of their documentation has been translated and is available on the Pi4J website. The original German documentation can be found on “CrowPi Goes Java.” For each component, the students have created a separate application. This makes it very clear how you can use their code as the foundation for your own project.
The proof-of-concept-application in this article uses some parts of the CrowPi Goes Java -project, which are refactored to fit into a stand-alone application to read from different sensors at the same time, or listen to their change events and publish data to HiveMQ Cloud MQTT broker.
The Pi4J library uses the modular approach. Building a project for the Raspberry Pi generates a directory with all the required files. That’s why, the pom-file can look a bit overwhelming.
The dependencies contain:
- The HiveMQ MQTT client to publish the data
- The Pi4J core and the plugins required for this project
- The SLF4J which is the logging framework used by Pi4J
- The Jackson and Jakarta.JSON for the JSON data generation to simplify the data transfer
Using multiple plugins, the following steps are taken:
- Compile the Java project
- Define the executable class
- Create the
jarin the distribution directory
- Copy a
run.shscript in the distribution directory
- Add the runtime dependencies in that same directory
The Code to Send Data
To make the code easy to understand, each functionality has been separated into its own class.
HiveMqSender.java is the main class where everything is initialized and the application started.
Publishing to HiveMQ MQTT Topics
In the file
HiveMqManager.java, we find all the code needed to connect to HiveMQ Cloud and publish messages.
We use an MQTT version 5 client which provides multiple improvements compared to version 3 as described on “Meet the New MQTT 5 Protocol - MQTT 5 Essentials Part 1” and the next articles in that series. And yes, you will also find out in that article, why there is no version 4.
Thanks to the builder-method we can very easily configure the client and connect. As HiveMQ Cloud requires a secure connection we need to use
sslWithDefaulftConfig() and apply the authentication in the connection.
To make it easy for our sensors to send data, we extend this class with a
sendMessage method. To see if our messages published successfully, or what went wrong, we use the
Reading the Data from the Sensors
All sensors are initialized in the
SensorsManager.java class. Each sensor is implemented as a
Component. Some of them are read with an interval with the
SendMeasurements started by a TimerTask. Others send changes through listeners, for instance
onNoise of the
How to Configure Pi4J?
As you can see in the previous class, the Pi4J Context is initialized with
CrowPiPlatform.buildNewContext(). Pi4J uses a plugin structure so it is easy to maintain and extend with additional functionality. In most cases, we can rely on
Pi4J.newAutoContext() for the initialization, but because we explicitly need the functionality of the PiGpio plugin, we use a helper class to make sure Pi4J loads the correct plugins. PiGpio is the native library (written in C) which is used under-the-hood by Pi4J to handle the GPIOs.
Take a look at the full sources of this project or the Pi4J CrowPi project if you are looking for a specific sensor.
In this post, we will take a look at one of the components: the sound sensor. This is a simple
DigitalInput sensor and all its code is implemented in
SoundSensorComponent.java. We only need to implement methods to handle the change of the pin between low (silent) and high (sound detected).
The pin is initialized with the following part of the code:
Pi4J also provides a builder-pattern to configure the GPIOs. For this sensor which is a
DigitalInput, we need to configure:
- address: the pin to which it is connected = BCM 24 on the CrowPi
- unique ID so Pi4J can handle this during its lifetime
- name: for our own reference, e.g. in logging
- pull up or down: how state changes need to be handled
- debounce: interval between state changes to avoid a too fast “ping-pong” between state changes
The CrowPi OS
You can run this project on any Raspberry Pi OS, which has Java. But, when you run it on a CrowPi, you can get a kickstart by using the CrowPi OS, which is part of the Pi4J project (again, thanks to the FHNW University).
A few of the sensors (distance and humidity) in the CrowPi require very strict timing to get reliable data results. This is very difficult to achieve on a Linux system, both with Python or Java. Some tweaking is possible in Linux to improve the results and these are provided by default in CrowPi Os, which is based on Raspberry Pi OS, but with some extra improvements for the CrowPi (thanks again to FHNW University).
But even with these tweaks, the results are not 100% reliable. These kinds of sensors are a better fit with a microcontroller like Arduino or Raspberry Pi Pico. A real time-operating system can also be used as described on Raspberry Pi Real-Time OS (RTOS): Which to Choose by Cherie Tan.
One last remark: this project uses the tilt sensor, which needs to be enabled with a dip switch on the CrowPi board:
Building and Running Java Application on Raspberry Pi
You can develop a Java project on your PC and copy the jar-files to the Raspberry Pi. Alternatively, copy the sources and compile on the board. You can also develop on the Raspberry Pi itself. Each has its own pros and cons. The Pi4J-website in the Getting Started section, has more info on each approach.
If you just want to run this example application, the easiest approach is probably to install Maven, download the sources, and build and run on the Raspberry Pi. If you want to take this quick-start, you can clone the code and run it in a few lines:
After some startup logging, you will see the scheduled messages and events of different sensors, e.g. below the motion and noise sensors events.
HiveMQ MQTT WebSocket Test Page
The logging of the application shows data is published to HiveMQ Cloud. We want to be sure before we continue to create a client. Luckily, there is a WebSocket client we can use to test it easily! Head over to HiveMQ WebSocket Client and fill in all the fields with your credentials. Make sure to select the SSL check box! When all the fields are filled correctly, click on Connect button. You will get a green Connected indication on the top of the demo page.
Now we can define the topics we want to subscribe to. For example, “crowpi/motion”, “crowpi/sensors.”
As soon as messages are pushed from the Raspberry Pi, they are shown in the web test page as shown below.
Great! The above image shows that the sensor data is flowing from our Raspberry Pi to HiveMQ Cloud!
Conclusion: MQTT on Raspberry Pi Made Simple with Java, HiveMQ Cloud, Pi4J & CrowPi
As always, when you combine Java with Maven, a project like this can be achieved with minimal code and still be easy to understand and manage. Pi4J adds the required tools to hide the complexity of dealing with GPIOs and ‘converts’ the electronic components to Java objects.
Thanks to the HiveMQ Cloud service, we can publish the data up to 100 devices to an always-on, maintenance-free MQTT message broker for free! Combined with the inexpensive Raspberry Pi, a world of opportunities is open for all makers.
In the next article of this blog series, MQTT on Raspberry Pi, I will show how to visualize the data of our sensors in a dashboard. And of course, again with Java (and JavaFX) on a Raspberry Pi! Stay tuned!