Taming AI Agents with context

The first wave of enterprise AI pilots created an illusion: that accuracy is something you get by picking the right model and writing a clever prompt. This belief has now collided with reality. The model doesn’t matter nearly as much as the context you place it in. And unlike prompts – which are basically persuasive essays – context has technical architecture, design choices, and measurable consequences.

This is what context engineering tries to formalize – an OS-layer most teams don’t realize they’re missing.

LLMs Have No Idea What You Want

LLMs can reason, but they cannot infer your policies, workflows, or priorities unless you place that information inside their working environment. They don’t know which document is authoritative, which number governs the calculation, which instruction overrides which exception. Without structure, they guess. And their guesses sound confident enough to make the issue look like an AI problem when it’s really an information-design problem.

Nothing in the model architecture fixes this. You can upgrade from “medium” to “extra-large” and still get wrong answers because your environment is noisy, ambiguous, or contradictory.

Why the Environment Matters More Than the Prompt

Every AI system operates inside a context window – a blend of instructions, data, memory, and workflow state. In most enterprises this window is created ad hoc: whatever retrieval system coughs up, plus a prompt someone wrote after lunch.

Context engineering replaces that improvisation with design. For a business reader, the easiest way to understand it is through analogy: it’s constructing the room in which good decisions are the only possible outcome.

Three technical foundations matter here:

1. What information is retrieved

2. How that information is structured

3. How the reasoning process is broken into steps

Everything else is an implementation detail.

1. Retrieval: The AI Needs the Right Data, Not More Data

Most corporate RAG implementations assume relevance = “vector similarity.” That works fine when the question is undemanding (“summarize this”). It collapses when precision matters. Similarity-based retrieval returns things that look related, not things that are correct.

The technical shift happening industry-wide is the move from “retrieve by vibes” to:

• Filtered retrieval – apply rules before embedding: Fetch only from authoritative sources; use only documents that match explicit metadata; consider only data that maps to the entity in the user query.

• Structured retrieval – break content into logical units: Sections, tables, field-level data, semantic components.

• Compositional retrieval – fetch from multiple systems depending on the sub-task, not depending on the overall question.

This is where context engineering first earns its keep. The question is no longer “How do we prompt the model?” but “How do we design retrieval so the model sees the right 3 KB instead of the wrong 300 KB?”

2. Structure: LLMs Behave Better When You Don’t Hand Them a Junk Drawer

Most enterprise contexts are a junk drawer: a few policy documents, a KB article, a blob of history, maybe a table or two. LLMs are generalists; they can parse mess, but they pay for that ability with uncertainty.

Context engineering organizes the world so the model doesn’t have to:

• Convert unstructured text into typed fields (“contract_value,” “policy_version,” “billing_cycle”) that the AI can reference explicitly – Schemas

• Mark certain pieces of data as authoritative and unrelated ones as disallowed – Canonical sources

• Clarify whether a given piece of context is a rule, an example, a definition, or an exception – Semantic roles

This dramatically improves reasoning, because LLMs are better logical engines when the inputs are well-typed and well-scoped.

In fact, Moative has a data orchestration product of its own that automatically builds semantic data clusters to serve the LLMs. This semantic structuring also allows you to control which contradictions the model sees. If the latest policy overrides the older one, the older one simply never enters context. Humans deal with this ambiguity all the time. LLMs don’t need to.

3. Process: Break Tasks Into Steps So the Model Isn’t Juggling Everything at Once

Enterprise tasks are multi-step by nature: interpret input → fetch data → validate → compute → apply policy → generate output. Most pilots compress all of this into one giant prompt. This is why the results feel like improv.

Context engineering pushes the system toward sequential reasoning architecture:

• Ask the model to identify the sub-task.

• Use that sub-task to retrieve specific context.

• Run the reasoning step in isolation.

• Evaluate intermediate output.

• Feed that into the next step.

There are many technical variations here – graph planners, hierarchical prompting, tool-calling loops – but the principle is simple: don’t make the model do everything at once.

Breaking the task into steps gives you auditability, repeatability, and the ability to govern behavior. You can fix one step without rewriting the whole system. You can enforce rules at the step level. You can evaluate each stage separately, instead of guessing which part failed.

Why This Matters for the Business Audience

Once you look under the hood, accuracy becomes less mystical. AI succeeds or fails for reasons you can measure. You can log exactly what data it retrieved, how it interpreted the instruction, which step produced the error, and which policy conflicted with which document.

Senior leaders don’t need to understand embeddings, token windows, or model architectures. But they do need to understand the implications:

• Accuracy is designed, not hoped for.

• Governance is a property of the workflow, not the model.

• Repeatability comes from structure, not clever prompting.

• Cost efficiency improves when the system retrieves less, not more.

This is also the only way to scale. Without context engineering, every new use case becomes its own snowflake – another custom prompt, another retrieval hack, another demo that doesn’t survive contact with production.

With context engineering, use cases become patterns: well-defined, replicable, governable. You get a consistent approach to correctness instead of a new art project every week.

The Pragmatic Version of the Story

Enterprises don’t need a PhD in NLP to benefit from this. They need to accept a simple premise: accuracy comes from environmental design.

Design retrieval thoughtfully. Structure context intentionally. Break tasks into steps. Record what the AI saw at each point. Test against real examples. Iterate.

The model is the least volatile part of the system. Context is the part you control, that determines whether AI feels like a junior analyst you trust or a very confident stranger making up numbers at the town hall.

Context engineering is simply the discipline of building that trust.

Enjoy your Sunday!

From our archives