Your AI System Can Explain Everything and Control Nothing

If you're deploying AI into any workflow that touches compliance, revenue, or similar decision areas, you're going to face a question soon, from your legal team, a regulator, a client, or a vendor: "Can this system explain its decisions?"

The instinct will be to say yes, and to invest accordingly. Explanation logs. Decision rationales. Audit trails. Your vendor will tell you their system is "fully explainable." Your compliance team will want documentation they can file. The EU AI Act, which is starting to shape procurement conversations even for US-based companies, explicitly requires transparency for high-risk AI systems. The pressure is real, the ask feels responsible, and the path forward seems obvious: build the explanation layer.

You should build it. But here's what will happen next, and it happens predictably enough that governance researchers have a name for it: explainability theater. An MIT Sloan and BCG expert panel coined the term this year after studying how organizations handle the relationship between explainability and oversight. A Jones Walker analysis in the National Law Review documented the legal version: companies satisfying the right-to-explanation with documentation that looks like transparency, while operators inside the organization don't actually understand how their systems reach conclusions.

The advice for avoiding this is straightforward. Monitor your system's aggregate behavior over time, not just individual decisions. Track drift. Assign ownership. Build escalation paths.

And yet, the pattern repeats. The interesting question is why.

The work that's visible to outsiders wins

When budget and attention are finite, organizations invest in the work that answers the questions being asked of them. Explainability pressure comes from outside: regulators, auditors, clients, board members. These people want artifacts. They want a document that says "here is why the system made this decision," and they want it in a format they can review, file, and reference later. Explanation layers produce exactly this. They generate deliverables that are legible to the people asking the hardest questions.

Monitoring produces something different. It produces internal operational awareness: dashboards, alerts, aggregate trend lines. This is valuable to the team running the system, but it's invisible to the people applying external pressure. No auditor asks "show me your drift detection." No client asks "who owns this system's behavior over time?" No regulator, at least today, asks "what happened the last time this system's output distribution shifted?"

So when the budget conversation happens, the explanation layer gets funded because it has a visible constituency. Monitoring gets deferred because it doesn't. The frameworks say "do both." The EU AI Act requires both decision-level transparency and post-market monitoring. The GAO's accountability framework is built around governance, data, performance, and monitoring; explainability isn't even one of the four pillars. The MIT Sloan expert panel found that 77% of experts believe explainability and human oversight reinforce each other.

The frameworks are right. But in practice, the organization does the one that produces a deliverable someone external can hold.

What the gap looks like at scale

Healthcare saw this play out in the starkest terms. Major insurers ran AI-driven prior authorization for years, generating per-decision rationales that satisfied procedural requirements. Each individual denial came with a formatted explanation. A Senate investigation surfaced that Medicare Advantage denial rates had risen 56% and that overturned denials on appeal exceeded 80%. The reasons behind that failure are complex and go well beyond monitoring gaps. But the observable pattern is relevant: individual explanations were procedurally complete while aggregate system behavior went unexamined.

For most companies, the gap surfaces less dramatically but no less painfully: a client notices inconsistent outputs, revenue numbers stop reconciling, a compliance review catches something that's been wrong for months.

Seeing it before it costs you

If you are thinking of implementing AI systems now, this is the pattern to watch for. The explainability investment will feel complete because it produces visible output. The monitoring investment will feel optional because nobody in the room is asking for it. Left unchecked, the gap between the two widens quietly until something external forces the question.

Three questions worth answering before you go live:

1. Who is responsible for this system's behavior over time, not for individual decisions, but for whether its aggregate output has shifted?

2. When this system's behavior changes, what happens? Who gets notified? What's the escalation path? What authority do they have to intervene?

3. What does this system know about its own past performance, and where does that information live?

If you can answer these clearly, you've built both halves. If you can't, you've built the visible one.

Enjoy your Sunday!

When Bharath Moro, Co-founder and Head of Products at Moative, explains guardrails using a salt shaker and tablecloth, guests usually take notes on their napkins.

More reading