Stuffing more context into a prompt is the cheapest thing an engineer can do and the most expensive thing a system can carry. We measured what happens when you compose instead.
The baseline: how much of a prompt is wasted
We instrumented a set of production agents at four companies for six weeks, capturing every context package sent to a model alongside the eventual answer. Then we asked: which items in the package actually contributed?
| Metric | Naive retrieval | Composed context |
|---|---|---|
| Avg. context tokens | 9,660 | 1,240 |
| Items in package | 18 | 3 |
| Items referenced in answer | 2.4 | 2.6 |
| Answer accuracy (graded) | 71% | 93% |
| P50 latency | 2.9s | 1.1s |
The headline number is the 87% token reduction. The more interesting number is the accuracy going up.
Why less context improves answers
Two mechanisms, both well documented and both underweighted in practice:
Distraction
Irrelevant but semantically adjacent content pulls the model toward plausible-sounding wrong answers. A stale churn definition sitting three chunks away from the current one is worse than no definition at all, because the model has no way to rank them.
Position degradation
Attention over long contexts is uneven. Content in the middle of a large prompt is measurably less influential than content at either end. Stuffing eighteen items guarantees that most of them land in the weak zone.
How composition works
Instead of ranking documents by similarity and taking the top N, we resolve the query against a structured knowledge graph:
- Intent classification — what kind of question is this? A metric lookup, a policy question, an entity definition, an open-ended analysis?
- Entity extraction — which concepts does it reference? Churn, cohort, Q3, enterprise tier.
- Graph traversal — pull the canonical definition for each entity, plus direct dependencies (churn depends on retention, which depends on active-account rules).
- Scope filter — drop anything outside the caller's permissions.
- Assembly — emit the minimal package with provenance attached.
What the 42ms is spent on
Composition adds a step, and people reasonably ask what it costs. Roughly: 8ms for intent and entity extraction, 19ms for graph traversal, 6ms for scope resolution, 9ms for assembly and serialisation. Against a model call measured in seconds, it disappears.
Caching note: intent classification results are cached by normalised query shape, and graph traversals are cached per entity set with invalidation on knowledge updates. In steady state most requests skip the first two steps entirely.
Where composition does not help
If the question is genuinely open-ended — "summarise everything we learned from the March incident" — there is no small canonical answer set, and retrieval over documents remains the right approach. Composition is for questions with structure. Fortunately, most enterprise questions have more structure than they first appear to.
Mahari Kalau
Writes about context engineering, agent architecture, and the unglamorous parts of enterprise AI.