Why do industrial AI teams still spend so much time on data preparation?

Ask a data scientist working on an industrial AI project where their time actually goes, and modeling is rarely the answer. It's cleaning sensor exports, chasing down what a field name means, reconciling two systems that both call something "temperature" but disagree on units. The model is often the fastest part of the project. Getting to a dataset a model can trust is the slow part and it stays slow project after project.

What makes industrial data so hard to prepare?

For decades, the priority was connecting more sources of data: more sensors, more historians, more systems reporting in. That problem is largely solved. Most industrial organizations aren't short on data access anymore. What they are short on is agreement about what the data means.

Operational data comes from systems that were never asked to agree with each other. A temperature reading from Line A and a temperature reading from Line B might use different units, different timestamp formats and no shared definition of what "temperature" refers to in either process. A batch ID in the manufacturing execution system might not match the batch ID in the lab system, even when they're describing the same production run. None of that gets resolved automatically. Someone has to reconstruct it by hand, and in most plants, that someone is a site engineer who happens to know the history.

That kind of tribal knowledge works, until it doesn't. 

It's slow to onboard, it doesn't scale across sites and it disappears when the person leaves. And it becomes a much bigger liability the moment an AI agent enters the picture. A person can call a colleague to ask what a field means. An agent can't. It needs the context already attached, or it will act on an assumption nobody checked.

Why does industrial data need so much reworking?

The pattern is consistent across industrial AI projects, even when the underlying systems differ:

1. Unlabeled units and formats. A value arrives without the metadata needed to interpret it correctly, so someone has to guess or go find out.

2. Inconsistent naming across systems. The same asset, batch or process step gets a different identifier in the MES, the historian and the quality system, and nothing maps them to each other.

3. Missing lineage. A number shows up with no record of where it came from, what transformations were applied to it or whether it passed any validation.

4. No shared definition of "trusted." Two teams pull what looks like the same metric and get different answers, because neither system flags data quality or confidence.

Each of these is a one-time fix if it's solved at the point where the data is created. Left unsolved, it becomes a recurring cost. Every new project that touches that data has to rediscover the same missing context, write its own patch and hope the next person maintains it.

How to fix the problem?

It's the natural first response: the data science team is overloaded with prep work, so add data engineering capacity. That treats the symptom, not the cause.

Every pipeline or transformation script written to patch missing context is one more thing to maintain. It has to be rebuilt the next time a source system changes, a sensor gets swapped or a plant adds a new line. The underlying work doesn't shrink. It moves further downstream and gets duplicated across every team solving the same interpretation problem in isolation, usually with a slightly different answer each time.

This is what makes the cost compound. 

A data-meaning problem handled per-project, per-team, doesn't average out to less work over time. It multiplies, because nobody is fixing the source.

The alternative is to govern and contextualize data closer to where it's created, not after it lands in a data lake or a model's training pipeline. That means giving data a consistent structure, a clear owner and quality indicators before it ever reaches the team that has to use it.

In practice, this looks like a small number of upstream decisions applied consistently:

• Connect data reliably from the systems that produce it, without requiring a full migration or a central repository first.

• Contextualize it at the point of connection: attach the asset, the process, the unit, the timestamp standard and a data-quality flag before it moves anywhere else.

• Analyze it and apply governance so all guardrails that are centrally defined, are consistently checked at each layer and enforced. 

• Act on it, including through AI agents, because the context and governance an agent needs to reason safely is already attached rather than assumed.

This is the premise behind HiveMQ's approach to industrial data: connect it reliably, then contextualize it so teams can analyze and act on it with confidence, rather than cleaning it after the fact, and hoping the next project inherits less mess than the last one.

What changes when context moves upstream?

None of this makes data preparation disappear. Real-world data will always need some judgment applied to it, and no architecture removes that entirely. What changes is the size of the job. Instead of every team rebuilding the same context from scratch, a data scientist inherits data that already knows what it is: which asset it came from, what unit it's in, whether it passed validation and who's accountable for it.

That's the difference between a data science team that spends its week on modeling and one that spends its week reverse-engineering someone else's spreadsheet. The remaining prep work shrinks from a full-time job to a reasonable one, and the hours that used to go into cleanup go into the work the team was actually hired to do.