Why Your Industrial Control System Is Broken

Introduction

Have you ever walked into a control room, looked around at everything running, and thought, “this whole thing feels harder to manage than it should”?

Nothing has completely failed. The equipment is running. Alarms are coming in. Operators are getting through the shift. But somehow, the whole system feels heavier than it should. Troubleshooting takes too long. Every change feels risky. People rely on memory more than documentation. Everyone is treading water.

After 36 years in industrial controls, I can tell you that feeling is almost always telling you something real. Most broken control systems do not become broken overnight. They become broken slowly through three things: configuration, architecture, and change. On their own, each one can make a system harder to live with. Together, they are behind most of the dysfunction I have seen in ICS and OT environments.

Broken Does Not Always Mean Down

When people hear “broken system,” they usually picture the obvious version. Lights out. Equipment stopped. Production halted. A major alarm on the screen. In industrial control systems, that is often the easy kind of broken, because at least everyone knows something is wrong.

The harder kind of broken is quieter. The system is technically operating, but it is difficult to troubleshoot. Similar areas behave differently for no obvious reason. Operators do not trust the alarms. Engineers are afraid to touch old logic. Drawings are questionable. Only one or two experienced people really understand how the pieces fit together.

This matters because industrial environments depend on more than uptime. They depend on maintainability, repeatability, and trust. A system that cannot be safely changed, clearly understood, or reliably supported is a business risk and, in many industries, a safety risk.

1. Configuration Drift Makes Everything Harder to Trust

A lot of systems are not suffering because of bad hardware. They are suffering because of poor and inconsistent configuration. Those are two very different problems.

Think about ten similar pumps across a facility. Same manufacturer. Same model. Five were set up by one integrator five years ago. The other five were set up by a different contractor two years later. Different tag naming. Different alarm limits. Different HMI faceplate behavior. Some bypasses were added during startup and never removed.

Now multiply that across 10 or 15 years of projects, contractors, programmers, and expansions. What you get is a system where nothing is quite predictable. Troubleshooting slows down because the technician cannot assume that similar assets behave the same way. Everything has to be verified from scratch.

Configuration drift is not just PLC programming style. It includes alarm limits that no longer match how operations actually runs, user accounts and permissions nobody has reviewed in years, HMI screens that look different from building to building, and inconsistent field device settings across equipment that should be identical.

2. Weak Architecture Turns Local Fixes Into Long-Term Complexity

Even good configuration cannot fully save a system with weak architecture underneath it. Configuration and architecture go hand in hand.

Here is the thing: many facilities did not start with bad architecture. They evolved into it. Project by project. Budget cycle by budget cycle. Expansion by expansion. A new vendor came in. A new platform solved a local problem. A temporary integration became permanent. A reporting tool got added because the historian could not quite do what the team needed.

At the time, each decision made sense locally. It solved a specific problem and got the project done. But over time, the facility ends up with too many platforms, too many vendors, poor separation between layers, and duplicated functions split between SCADA, historians, PLCs, and third-party tools.

Architecture problems usually do not hurt right away. That is what makes them dangerous. They show up later as complexity, brittleness, rising lifecycle costs, and upgrade projects that take far longer than expected because the existing system is so tangled.

3. Unmanaged Change Erodes Good Systems

This is where a lot of good systems start to fall apart. Change is constant in industrial environments. Equipment gets upgraded. Processes shift. New reporting requirements come in. Security expectations change. You are never going to stop change from happening, and you should not try to.

The problem is not change itself. The problem is uncontrolled, undocumented, poorly governed change.

Emergency edits get made directly in production under pressure. Nobody has time to write them up, so they do not get documented. Drawings never get updated. PLC logic is modified, but backed up incorrectly. Contractors come in, do the work their way, and leave. Your team does it another way. Neither approach gets captured clearly enough for the next person.

This is why management of change has to be more than a compliance form. It needs to be a real operating discipline. Changes should be reviewed, tested, documented, backed up, and reflected in drawings and records as part of the work, not as something everyone hopes to clean up later.

Configuration, Architecture, and Change Reinforce Each Other

These three problems do not stay in their own lanes. They feed each other.

Weak architecture makes configuration harder to standardize. If the facility has fragmented platforms and no clear design intent, there is no solid foundation for consistency. Configuration drift almost becomes inevitable.

Poor configuration makes every future change riskier. If you cannot trust the baseline, every modification becomes an investigation. You are not just making a change, you are trying to figure out what the current system really is.

Unmanaged change slowly erodes both configuration and architecture. Every undocumented edit makes the baseline less trustworthy. Every project that bolts something on without checking the architecture makes the overall design a little weaker.

What This Looks Like in a Real Facility

When drift has taken hold, the symptoms are usually familiar. Operators do not trust the alarms because there are too many nuisance alarms or alarms that have been wrong for too long. Maintenance staff work around the system instead of with it. Engineers avoid old code because nobody knows what it does.

Similar assets behave differently for no obvious reason. Drawings are questionable. Upgrades take twice as long as they should because the first two weeks are spent figuring out what is actually there. Troubleshooting depends on one veteran person who carries the whole system in their head.

What You Can Actually Do About It

Let me be clear: I am not talking about tearing everything down and starting over. That is rarely practical, and it is often not necessary. The real goal is to regain control of the system lifecycle gradually and deliberately.

One of the best questions you can ask during any project review is simple: does this decision move us toward our target architecture, or further away from it? That question alone changes the conversation. It forces the team to think beyond the immediate project and into the long-term health of the control system.

The Bottom Line

Industrial control systems usually do not become broken because of one device, one bug, or one bad decision. They become broken gradually through configuration drift, weak or fragmented architecture, and unmanaged change.

If your system feels harder to maintain than it should, ask yourself where the real issue is. Is it configuration? Is it architecture? Is it change? Or, as is often the case, is it all three reinforcing each other?

The good news is that you do not have to fix everything at once. Start by regaining control of the lifecycle. Standardize what you can. Define where the architecture is heading. Make change control real. Treat technical debt as something you manage before it turns into failure.