Connect: Building a real-time data backbone for data accessibility

Agentic AI promises to transform industrial operations by continuously monitoring production, identifying inefficiencies, recommending corrective actions, and coordinating workflows across people and systems. But an AI agent can only understand and influence operations when it has reliable access to what is happening on the plant floor in real time.

This is the purpose of the Connect stage: to onboard your machines, applications, databases and industrial systems into a common data-streaming infrastructure that makes operational information available in real time.

This blog series and whitepaper presents a four-stage approach for building a scalable data foundation for real-time operational intelligence:

• Connect makes operational data available in real time.

• Contextualize makes it understandable, governed, and trustworthy.

• Analyze turns it into actionable information.

• Act enables trusted systems and people to respond safely and consistently.

Together, these stages create immediate value for real-time visibility, standardized KPIs, operational analytics, quality monitoring and decision support across the enterprise. They also establish the event-driven and semantic foundation required when the organization is ready to delegate selected, clearly governed tasks to AI agents.

The goal here is to help you build an architecture that improves operational performance today, through operational visibility and data-driven optimization, while creating a controlled path toward more intelligent and agentic industrial operations over time.

The objective of the Connect stage is not simply to collect more data. It is to establish an enterprise-wide backbone through which systems can exchange events, state changes, measurements and business information without requiring a new point-to-point integration every time you add an application, machine, production line or factory.

Your real-time data backbone should make it easier to connect what already exists, introduce new technologies and scale successful use cases across your manufacturing enterprise.

Move beyond point-to-point data integration

Most manufacturing environments contain a combination of modern platforms, legacy systems, industrial equipment, databases and vendor-specific applications.

Depending on your operations, for example, in life sciences manufacturing you may need to connect:

• PLCs and OPC UA servers

• IoT gateways and connected devices

• Industrial connectivity platforms, such as Kepware

• Distributed control systems (DCS), such as Emerson DeltaV

• Data historians, such as AVEVA PI

• Manufacturing execution systems (MES), such as Siemens Opcenter

• Laboratory information management systems (LIMS)

• Computerized maintenance management systems (CMMS)

• Enterprise resource planning systems (ERP)

• SQL databases and flat files

These systems were typically introduced at different times, for different purposes and by different teams. They use different interfaces, communication protocols, information models and update frequencies.

Connecting them through individual point-to-point integrations may work for a limited use case, but this approach becomes increasingly difficult to sustain as your architecture grows.

Every new consumer requires another connection. Every system change can affect several downstream applications. A successful pilot may take weeks or months to reproduce at another site because the integration work must be repeated.

Your architecture gradually becomes a tightly coupled network of custom interfaces that is expensive to maintain and difficult to change. You therefore need to separate the systems that produce operational data from the systems that consume it.

How does MQTT decouple your industrial data producers and consumers?

A publish-subscribe architecture allows your operational systems to share data without needing to know which applications will use it. Producers publish events and state changes to a common data-streaming layer. Consumers subscribe to the information they need.

For example, a production system may publish a change in equipment status. That event could be consumed simultaneously by:

• An MES system updating the current production state

• A maintenance application evaluating equipment health

• A dashboard displaying line performance

• A data platform storing information for long-term analysis

• An AI agent identifying abnormal operational behavior

The production system does not need a separate integration with each of these applications. It publishes the event once, and the data-streaming infrastructure distributes it to the appropriate consumers.

This decoupling changes how you scale your digital architecture. You can add new consumers without continually modifying the systems that generate the data. You can also replace or upgrade a consuming application without redesigning the original machine connection.

MQTT is particularly well suited to this model because it provides a lightweight, event-driven publish-subscribe mechanism designed for reliable and scalable data distribution.

Rather than treating your data backbone as another centralized application, you use it as a shared communication layer through which operational systems exchange information.

Translate industrial protocols into a common data-streaming layer

Your machines and operational systems will not all communicate using MQTT directly. Many will expose data through protocols and interfaces such as OPC UA, industrial Ethernet protocols, database connections, REST APIs, vendor-specific connectors, or file exchanges.

Your architecture therefore needs a protocol translation layer.

At the edge, connectivity software or an industrial gateway connects to the native interfaces used by your machines and systems. It reads the available data and translates it into a common format that can be published through your data-streaming infrastructure.

In many plants, machine and control-system data is first aggregated or normalized through an OPC UA server. An edge gateway then connects to that server, retrieves the relevant data, and publishes it to MQTT.

The same architectural principle applies to information coming from MES, LIMS, CMMS, ERP, historians, databases and other applications.

Protocol translation allows you to bring legacy systems into your digital infrastructure without requiring every device or application to support the same native protocol.

It also prevents downstream consumers from needing separate connectors for every source system. Consumers can access operational information through the common data-streaming layer instead of integrating directly with every machine, database and enterprise application.

Build a manufacturing data backbone for events and state transitions

Your data backbone should not be designed only to move continuous streams of sensor values. Manufacturing operations are driven by events and state transitions:

• A machine changes from running to stopped

• A batch moves into a new process phase

• A work order is released

• A quality sample becomes available

• A laboratory result is approved

• A maintenance notification is created

• A material is consumed

• A process parameter exceeds its expected range

• An operator acknowledges an alarm

These events describe what is happening across your operations. They connect physical production activity with the systems responsible for quality, maintenance, planning and business execution. Your data-streaming layer should therefore support the exchange of both time-series measurements and operational events.

In a pharmaceutical production scenario, for example, process values from a reactor may need to be associated with batch events from the MES, laboratory results from the LIMS, equipment history from the historian, and maintenance information from the CMMS.

The Connect stage does not yet resolve all of these relationships. That occurs during contextualization. However, it must ensure that the required data and events are available, timely, traceable and accessible through the same underlying infrastructure.

Without that connectivity, it is difficult to assemble a complete digital batch record, investigate a quality deviation or provide an AI agent with the evidence needed to support an operational decision.

Use a site-to-enterprise architecture

Your architecture should support local manufacturing operations while also making selected information available across the enterprise.

A practical implementation can follow this flow:

1. Generate and aggregate the industrial data

Data originates from machines, sensors, laboratory instruments, controllers, production systems, databases and enterprise applications.

At the plant level, equipment data may be aggregated through PLCs, DCS platforms, historians or OPC UA servers.

2. Translate and prepare the data at the edge

Edge connectivity software connects to the source systems using their native interfaces.

It translates the data into a format suitable for distribution through MQTT. Basic source information, timestamps, quality indicators and other metadata may also be added at this stage.

The edge layer should remain focused primarily on connectivity, normalization and reliable transport. More advanced semantic modelling and enterprise context belong in the next stage of the architecture.

3. Distribute information locally

The edge layer publishes operational information to an on-premises MQTT broker.

The site broker provides a local publish-subscribe layer for plant systems and applications. Local consumers can continue to access critical information even when connectivity to a central data center or cloud platform is interrupted.

This supports low-latency communication and reduces unnecessary dependence on wide-area network connectivity.

4. Bridge selected information to the enterprise

The site broker forwards approved topics to a central MQTT broker hosted in a data center or cloud environment.

Broker-to-broker bridging allows you to move operational information between sites and enterprise platforms without requiring each plant system to connect directly to the cloud.

You can control which information remains local, which information is shared centrally, and how data flows between different environments.

5. Connect plant OT, plant IT and enterprise consumers

Once information is available through the backbone, plant and enterprise systems can exchange events through a common connectivity infrastructure.

OT systems such as PLCs, DCS platforms, gateways, and historians can share information with IT systems such as MES, LIMS, CMMS, ERP, analytics platforms, data services and AI applications.

This creates the foundation for a more integrated operational environment without forcing every system into a direct dependency on every other system.

Design for enterprise scale

A data backbone that works for one production line may not be sufficient for an enterprise deployment. Your architecture should be designed to support increasing numbers of devices, topics, applications, users, sites and data flows.

At enterprise scale, your MQTT infrastructure should be able to:

• Support millions of topics and large numbers of concurrent client connections

• Distribute connections and workloads across broker nodes

• Continue operating when an individual node becomes unavailable

• Retain and restore client sessions where required

• Route information efficiently between sites, data centers and cloud environments

• Prevent slow or unavailable consumers from disrupting data producers

• Scale without forcing you to redesign the application architecture

Load balancing can help distribute client connections evenly across your broker cluster. Clustering and high-availability mechanisms can prevent a single broker instance from becoming a central point of failure. These capabilities matter because your data backbone will increasingly support operationally important workflows.

If the data pipeline becomes unavailable, you do not only lose a dashboard. You may also lose visibility into production, quality, maintenance, or material movement. In modern manufacturing, availability of operational data is part of operational resilience.

Treat security as an architectural requirement

Connecting more systems increases the number of potential pathways through which your operational information can be accessed. Security must therefore be built into the data-streaming architecture rather than added after the connectivity layer has been deployed.

Your backbone should support end-to-end TLS encryption so that data is protected while moving between clients, gateways, brokers, and enterprise platforms.

It should also support the authentication methods required by your environment, which may include:

• Username and password authentication

• Client certificates

• Enterprise identity providers

• Token-based authentication

• Integration with centralized identity and access-management systems

Authorization should determine which clients can publish or subscribe to specific parts of the topic structure.

A maintenance application, for example, may be permitted to consume equipment-health information but not sensitive production or laboratory data. A site gateway may be allowed to publish only within the topic hierarchy assigned to its factory.

Centralized security policies help you maintain a consistent security posture across sites and data flows. However, your architecture may also need delegated site-level controls to reflect local ownership and operational responsibilities.

The objective is to make data broadly accessible to authorized users and systems without making it indiscriminately available.

Establish data governance without overloading the connectivity layer

Your Connect architecture needs enough governance to keep data flows reliable and manageable.

At a minimum, you should define:

• Who owns each source connection

• Which systems are authoritative for particular data

• How topics are named

• Which payload formats are permitted

• How schema versions are managed

• Which information can leave the plant

• Which consumers are authorized to access it

• How connections, certificates and credentials are managed

• How data quality and delivery failures are monitored

Standardized schemas and interface contracts can reduce downstream integration work. Some consumers may be able to ingest MQTT payloads directly, avoiding unnecessary transformations and simplifying the data pipeline.

However, you should avoid attempting to solve the entire information-modelling problem inside the connectivity layer.

The Connect stage should make data consistently available. The Contextualize stage will give that data richer meaning by applying shared asset structures, business context, semantic relationships, governance and operational models.

Maintaining this separation of responsibilities keeps your architecture easier to evolve.

Design for change, not solely for today’s use case

The most important measure of your integration architecture is not whether it supports your first use case. It is whether it reduces the effort required to implement the next one.

You need to be able to introduce a new analytics platform, maintenance application, AI service or production system without rebuilding your underlying integrations. You also need to bring older equipment and legacy applications into your digital infrastructure without waiting for a complete plant modernization program.

If every new deployment requires weeks of custom engineering, your pilots will remain isolated. Your teams will spend more time recreating interfaces than improving manufacturing operations.

Flexibility is therefore not an optional architectural quality. It is what allows your digital infrastructure to evolve at the pace of your business and technology landscape.

A decoupled data-streaming backbone gives you that flexibility. It allows data producers and consumers to change independently, supports gradual modernization and provides a common access layer through which operations, maintenance, quality, engineering, analytics and AI teams can use real-time information.

What you should have at the end of the connect stage

By the end of this stage, you should have:

• A scalable, highly available MQTT-based data-streaming layer

• A repeatable method for connecting machines, control systems, databases and industrial applications

• Edge components that translate native protocols into a common streaming format

• Local site brokers supporting plant-level communication

• Controlled broker bridging between factories and enterprise environments

• Secure authentication, encryption and topic-level authorization

• Clear ownership and governance for connections, schemas and data flows

• Real-time access to the events and state transitions required by downstream applications

You have not yet created a complete semantic representation of your manufacturing operations. A temperature value may be available, but a consuming system may not yet understand which asset produced it, which production order it relates to, which material was being processed, or how it affects quality.

That is the next architectural challenge.

Once you have built the real-time data backbone, you can begin to contextualize your connected data, turning isolated values and events into trusted, interoperable operational information that your people, applications, analytics and AI agents can understand.

Discover all four stages of building an architecture that improves operational performance today, in the full whitepaper: Building a scalable data foundation for real-time operational intelligence whitepaper.