How to Use MQTT and HiveMQ to Build a Chat Tool with Integrated Translation
MQTT is known to be the de facto standard for the Internet of Things as it is a lightweight, robust, and reliable protocol. MQTT implements the Publish/Subscribe pattern which can be demonstrated very easily, e.g. with a simple chat application. Users can send messages and receive them by subscribing to certain topics. But MQTT has many more features that, in conjunction with the HiveMQ Broker, can be used to build powerful solutions. Imagine having a chat application with automatic translation into the recipient’s language while the sender just sends messages in his native language. Let’s have a look at how this can be implemented with MQTT and HiveMQ and how much effort it takes.
MQTT User Properties
In MQTT 5, User Properties fundamentally operate as straightforward UTF-8 string key-value pairs. Their utility lies in their ability to be affixed to nearly every category of MQTT packet, with the sole exceptions of PINGREQ and PINGRESP. This broad application extends to various control packets like PUBREL and PUBCOMP.
The power of User Properties lies in their uncapped potential — as long as the maximum message size isn’t exceeded, you are free to employ an infinite number of User Properties. This opens up vast possibilities for enriching MQTT messages with additional metadata, facilitating a fluid transmission of information between the publisher, broker, and subscriber.
As our Chat Application should automatically translate messages into the recipient's language, we need to find a way to add this information to individual client connections. Given that we already have User Properties, it can be used to enrich MQTT messages with additional data as key-value pairs. Luckily, User Properties can also be used with the CONNECT packet offering the possibility for clients to specify their preferred language when connecting to the broker.
Here you can see how this is done using the HiveMQ MQTT CLI:
❯ mqtt sh
mqtt> con -Cup chat-language=en -i en-client
mqtt> sub -t mychat -ocThis will connect to the MQTT broker with the user property “chat-language” set to “en” and thereby defines that the user wants to receive messages in the English language. In this example, we are using ISO 3166 alpha-2 country codes.
Now that we also want to receive messages, we need to subscribe to a topic. In this case “mychat” is used as the “channel” name for chat messages.
This is a very good example of how to use MQTT 5 User Properties to transmit meta information with a MQTT message. As described previously, User Properties can be used with nearly every category of MQTT packet.
In the next step we will learn how to use the HiveMQ Extension SDK to make use of these User Properties and do the translation of messages.
HiveMQ Extension SDK
The flexible HiveMQ extension framework provides an open API that allows developers to create custom extensions that suit their specific infrastructures. The HiveMQ extension framework gives you the ability to seamlessly augment HiveMQ broker functionality with custom business logic.
In this example, we will use the framework to integrate HiveMQ with an external system to do the translation. There are a lot of services that can be used for that. We decided to use DeepL as it provides an HTTP API and offers a free tier.
Getting started with extension development is really straightforward. HiveMQ offers a very easy-to-follow quick start guide that describes how to set up your project, use the HiveMQ Gradle plugin, and develop your extension. The guide also provides good insights on how to test and debug your extension.
HiveMQ Community Extension SDK
The open HiveMQ extension framework adds many types of functionality:
Intercept and manipulate MQTT messages
Integrate other services
Collect statistics
Add fine-grained security
and much more
For more information, see HiveMQ Community Extension SDK Services.
So for our use case, we basically need to do three things:
Get the language user property from the CONNECT packet when a client connects
Attach it as an attribute to the connection
Translate a message before it is published
Getting MQTT User Properties on CONNECT
There are actually two different ways we could get user properties from a CONNECT packet:
Using the
ConnectInboundInterceptorinterfaceUsing the
ClientLifecycleEventListenerinterface
For this use case we will be using the ClientLifecycleEventListener as we do not need to modify the CONNECT packet and might also want to handle other client life cycle events such as disconnect or successful authentication.
public class ChatTranslationClientListener implements ClientLifecycleEventListener {
private static final @NotNull Logger log = LoggerFactory.getLogger(ChatTranslationClientListener.class);
private static final String CHAT_LANGUAGE = "chat-language";
@Override
public void onMqttConnectionStart(final @NotNull ConnectionStartInput connectionStartInput) {
// get the connect packet
final ConnectPacket connectPacket = connectionStartInput.getConnectPacket();
if (connectPacket.getUserProperties().isEmpty()) {
return;
}
// get user property for chat language
final Optional<String> chatLangPropOpt = connectPacket.getUserProperties().getFirst(CHAT_LANGUAGE);
chatLangPropOpt.ifPresent(chatLang -> {
// get the connection attribute store
final ConnectionAttributeStore connectionAttributeStore = connectionStartInput.getConnectionInformation().getConnectionAttributeStore();
// put chat language to connection attribute store
connectionAttributeStore.put(CHAT_LANGUAGE, ByteBuffer.wrap(chatLang.getBytes(StandardCharsets.UTF_8)));
});
}
@Override
public void onAuthenticationSuccessful(final @NotNull AuthenticationSuccessfulInput authenticationSuccessfulInput) {
// ignored
}
@Override
public void onDisconnect(final @NotNull DisconnectEventInput disconnectEventInput) {
log.info("Client disconnected with id: {} ", disconnectEventInput.getClientInformation().getClientId());
}
}You can use the onMqttConnectionStart method to get access to the CONNECT packet and check for User Properties.
Store the Language in the Connection Attribute Store
If the client transmitted a property with the key “chat-language” we store its value to the ConnectionAttributeStore. The Connection Attribute Store is a key-value store that preserves data as additional information in the MQTT client connection. All data is stored in memory. The maximum size of a single key-value pair is 10 kilobytes.
The Connection Attribute Store is useful for storing temporary data for a connected MQTT client or data that you want to clean up automatically after the client disconnects. The Connection Attribute Store is also useful for storing temporary information that you want to share across callbacks.
Translate Messages Before Publish
Now we have all the required information in place and know for each client in which language he wants to receive messages. But so far, no translation is done, and we need to find a proper place to implement it.
As we actually want to modify the PUBLISH packets before they are sent to the individual clients, using the PublishOutboundInterceptor interface is the correct choice as it allows us to do so by implementing the onOutboundPublish method. As most interceptor callback methods in the HiveMQ Extension SDK API, the onOutboundPublish method contains two parameters:
The first parameter is a read-only
PublishOutboundInputobjectThe second parameter is a modifiable
PublishOutboundOutputobject
Read more about the Extension Input / Output Principles here.
Calling External Services in HiveMQ Extensions
Before we dive deeper into the implementation of the onOutboundPublish method, there is one important thing to understand. The single most important rule for all extensions is: Never block an output! What does that mean?
For our translation, we need to call an external service which is a rather costly operation with the risk of timing out; this could affect the broker’s overall performance. Luckily, there is a way of getting around this and preventing possible blocks by using asynchronous output in conjunction with the ManagedExtensionExecutorService.
@Override
public void onOutboundPublish(@NotNull PublishOutboundInput publishOutboundInput, @NotNull PublishOutboundOutput publishOutboundOutput) {
// get the modifiable PUBLISH packet
final ModifiableOutboundPublish publishPacket = publishOutboundOutput.getPublishPacket();
if (!"translate".equals(publishPacket.getTopic())) {
return;
}
// get the original payload from read-only PUBLISH packet
Optional<ByteBuffer> untranslatedPayloadByteBuffer = publishOutboundInput.getPublishPacket().getPayload();
if (untranslatedPayloadByteBuffer.isEmpty()) {
return;
}
// try to get language attribute from Connect Attribute Store
final ConnectionAttributeStore connectionAttributeStore = publishOutboundInput.getConnectionInformation().getConnectionAttributeStore();
final Optional<String> languageOpt = connectionAttributeStore.get(CHAT_LANGUAGE).map(byteBuffer -> StandardCharsets.UTF_8.decode(byteBuffer).toString());
languageOpt.ifPresent(language -> {
LOG.info("Performing translation to [{}]", language);
// make the output async with a timeout of 2 seconds
final Async<PublishOutboundOutput> async = publishOutboundOutput.async(Duration.ofSeconds(2));
// call the translation service which will call the external DeepL API
final CompletableFuture<String> translationFuture = translationService.translate(untranslatedPayloadByteBuffer.get(), language);
translationFuture
.whenComplete((translatedPayload, throwable) -> {
if (throwable != null) {
LOG.error("Error translating: {}", throwable.getMessage());
return;
}
LOG.info("Translated: '{}'", translatedPayload);
// set the payload of the PUBLISH packet with the translated text
publishOutboundOutput.getPublishPacket()
.setPayload(ByteBuffer.wrap(translatedPayload.getBytes(StandardCharsets.UTF_8)));
//Always resume the async output, otherwise it will time out
async.resume();
});
});
}
Putting it All Together
Finally, it is time for testing! As you can see in the screenshot below we have four clients connected to the HiveMQ broker. Each setting the “chat-language” User Property to a different value resulting in messages being translated in the corresponding language. Nice!
Conclusion
Chat or messenger applications may not be the typical application for MQTT even though some big players are using it, like Facebook Messenger. It is far more used in the IoT and IIoT domain but certainly not limited to it.
The primary intention of our little example is to demonstrate the power of MQTT and the HiveMQ Platform. It can easily be transferred to other use cases in other spaces like automotive, manufacturing or logistics as well. And it shows how easy you can build complex solutions to your specific needs by leveraging MQTT features and the HiveMQ Platform with its extendability.
Sven Kobow
Sven Kobow is part of the Professional Services team at HiveMQ with more than two decades of experience in IT and IIoT. In his role as IoT Solutions Architect, he is supporting customers and partners in successfully implementing their solutions and generating maximum value. Before joining HiveMQ, he worked in Automotive sector at a major OEM.
