Prefill, Not Decode, Is Your Agent's Real Bottleneck

Your agent’s p95 latency just crept up again, and the dashboard says decode throughput is fine — tokens-per-second hasn’t moved. So the team looks at batch size, speculative decoding, quantization, the usual decode-side levers. None of it touches the number the user actually feels: time-to-first-token. That’s because the request isn’t slow because the model is generating tokens slowly. It’s slow because before generation starts, the model has to read a multi-thousand-token pile of retrieved documents, conversation history, tool outputs, and system prompt — and that pass, prefill, is a completely different computational job than the one your tuning knobs were built for.

This is the failure mode most teams don’t have instrumentation for. Prefill is compute-bound: a single dense forward pass over the entire input, dominated by matrix multiplication and bounded by FLOPS. Decode is memory-bound: one token at a time, dominated by how fast you can move KV cache through memory bandwidth. They are, in practice, two different workloads running on hardware built to do both adequately rather than either well. As context windows grow, RAG pipelines retrieve more, and agentic workflows chain more model calls — each one inheriting the accumulated context of everything before it — the share of your GPU fleet’s time spent on prefill keeps climbing, and nobody notices until time-to-first-token has quietly doubled.

Why This Is Showing Up Now, Not Two Years Ago

Single-turn chat with a short prompt has a trivial prefill cost — a few hundred tokens, done in milliseconds, invisible next to decode time. That’s why prefill was ignorable for so long. Three things changed it. First, context windows: 128K–1M token windows are now routine, and a document-heavy RAG retrieval or a long codebase context can push a single request’s prefill into the tens of thousands of tokens. Second, retrieval itself: every RAG call means the model reads retrieved chunks in full before it can say a word, and that read is pure prefill. Third, and biggest, agentic workflows: a single user request can now trigger five, ten, or twenty sequential model calls, each one re-reading accumulated context — tool outputs, prior reasoning, retrieved state — from scratch unless something is actively caching it.

The industry’s response has been disaggregation: split prefill and decode onto separate GPU pools instead of asking one fleet to time-slice between two workloads with opposite resource profiles. NVIDIA Dynamo, vLLM, SGLang, Ray Serve LLM, and LMCache all now ship disaggregated serving as a supported mode, and the throughput numbers are not subtle — NVIDIA has published roughly 30x higher served-request counts for large MoE models on GB200 NVL72 with disaggregated FP4 serving versus aggregated serving, and over 2x for Llama-class models on Hopper. Meta, LinkedIn, and Mistral are running disaggregated vLLM in production today, not as a research exercise. Separately, a September 2026 vendor analysis from optical-compute startup Lumai put numbers on the underlying imbalance: as effective inference compute demand scales roughly 1,000x over the next five years, delivering that on conventional accelerators alone would require infrastructure and power investment in a range that makes prefill efficiency a first-order economics question, not a tuning afterthought. Take the specific multi-trillion-dollar projection with the skepticism due any vendor’s five-year TAM claim, but the direction — prefill inefficiency becoming a binding constraint on what agentic and long-context applications you can afford to run — is showing up across independent framework benchmarks, not just one company’s white paper.

The Metric Your Dashboards Are Probably Missing

If your observability stack reports tokens-per-second and p95 end-to-end latency but not time-to-first-token as a distinct, alerted metric, you have a blind spot exactly where agentic workloads are most likely to break. TTFT is the sum of queueing time plus prefill compute time; under load, it’s queueing that spikes first, because a handful of long-context requests can monopolize a GPU’s compute for the seconds it takes to prefill them, starving shorter requests behind them in the queue. This is the head-of-line blocking problem, and it’s invisible in aggregate throughput numbers because the GPU looks busy and productive the whole time — it’s just busy on the wrong metric for what the user is waiting on.

Architecture Impact

What changes in system design? Serving architecture splits into two pools with different scaling rules: a compute-optimized prefill tier sized for peak concurrent context volume, and a memory-bandwidth-optimized decode tier sized for concurrent generation streams. KV cache transfer between the two tiers becomes a first-class network problem — NIXL, RDMA, or equivalent low-latency transport is now part of the inference stack, not an implementation detail. Routing also changes: requests need to be scheduled with prefill cost estimated up front (roughly proportional to input token count) so a scheduler can avoid parking short requests behind long ones.

What new failure mode appears? Head-of-line blocking at the prefill tier: a burst of long-context or multi-document RAG requests spikes queueing time for every request behind them, and end-to-end latency degrades even though decode throughput metrics look completely normal. A second failure mode is cache-cold agentic chains — if KV cache from step N of an agent loop isn’t reused at step N+1, the system re-prefills the same accumulated context repeatedly, multiplying compute cost roughly linearly with chain length for no generation benefit.

What enterprise teams should evaluate:

• Platform/infra team: whether your serving framework (vLLM, SGLang, Dynamo, Ray Serve LLM) supports disaggregated prefill/decode and prefix-cache reuse across agent turns, not just within a single request
• SRE/observability team: whether TTFT is tracked as a distinct SLO, separately alerted from tokens-per-second and end-to-end p95
• FinOps/platform-cost team: what fraction of current GPU-hours is prefill versus decode, since that ratio determines whether disaggregation pays for itself at your scale
• Application/agent teams: whether agent loops pass full conversation history on every call or use incremental context plus cache reuse, since this is the single biggest lever on prefill cost in agentic pipelines

Cost / latency / governance / reliability implications: Disaggregation delivers its throughput gains by adding operational surface area — two pools to capacity-plan, a cache-transfer network to keep healthy, and a router that has to make correct prefill-cost estimates or you reintroduce the imbalance you were trying to fix. Framework-reported gains cluster in the 2x–30x served-request range depending on model size and hardware generation, but that’s throughput under disaggregation done well, not a number that transfers automatically; teams that disaggregate without prefix caching or with a naive router often see marginal gains that don’t justify the added complexity. On governance, TTFT SLOs matter disproportionately for agentic and interactive use cases, since a user or downstream agent step waiting on a slow prefill has no visibility into decode-side performance being fine — the failure is felt entirely at the front of the response.

Implementation Guide

Start by measuring before you architect. Instrument time-to-first-token as its own metric, broken out by request context length, and look at its distribution under your actual peak load — not average load, since prefill contention is a burst phenomenon. If your p50 TTFT is fine but p99 is multiples worse specifically during traffic spikes, you’re looking at head-of-line blocking, and that’s a scheduling problem before it’s a hardware problem. Most teams find this out is the actual issue only after they’ve already spent a quarter tuning decode-side batching that never moved the number that mattered.

The high-leverage starting point isn’t buying specialized prefill hardware — it’s prefix caching and disaggregated serving on infrastructure you already have. vLLM’s automatic prefix caching and SGLang’s RadixAttention both let you reuse KV cache across requests that share a prefix, which is exactly the pattern in agentic loops (same system prompt, same tool schema, incrementally growing conversation) and multi-turn RAG (same retrieved document set across several follow-up questions). Turning this on is usually a config change, not a rearchitecture, and it’s the single highest-ROI move available before you touch hardware topology at all. Only after you’ve confirmed prefix caching isn’t enough — because your workload genuinely has high cache-miss rates, like unique long documents per request — does splitting into dedicated prefill and decode pools via Dynamo, vLLM’s disaggregated mode, or Ray Serve LLM become the next step.

The common mistake is treating this as a decode-tuning problem because tokens-per-second is the metric everyone already has dashboards for. Teams spend real engineering time on speculative decoding and continuous batching while TTFT quietly degrades, because nobody set up the alert that would have caught it. The second common mistake is disaggregating prematurely — standing up separate prefill and decode pools before confirming, with actual measurement, that prefill is your bottleneck and not, say, an under-provisioned decode tier or a slow retrieval step upstream of the model call entirely. Disaggregation adds a KV-cache transfer network and doubles your capacity-planning surface; it should be a response to measured contention, not a default architecture choice.

You’ll know it’s working when TTFT p99 stops spiking under load bursts and stays proportional to context length rather than to queue depth, and when your GPU-hours-per-request for agentic chains drops after you add cache reuse — that second number is the one that shows up on the infrastructure bill. If you disaggregate, watch KV cache transfer latency between pools as its own metric; if it creeps up, your network fabric, not either compute tier, is now the constraint.

The six-to-twelve month maturity path looks like this: teams start by adding TTFT as a first-class SLO and turning on prefix caching, which typically recovers most of the easy wins with minimal architectural change. Over the next few months, teams with genuinely high-prefill workloads — long-document RAG, large codebases, high agent-chain depth — move to disaggregated serving on their existing GPU fleet, which is where the framework-reported 2x–30x throughput gains actually materialize. Teams furthest along start treating prefill cost as an input to product decisions: capping agent chain depth, compressing retrieved context before it hits the model, or pricing long-context features to reflect their actual compute cost rather than treating all tokens as equivalent. The teams that skip straight to specialized hardware without first fixing caching and scheduling tend to buy capacity they didn’t need to.

Sources

• The Next AI Infrastructure Challenge Is Before the First Token
• Disaggregated Serving — NVIDIA Dynamo Documentation
• Disaggregated Prefill-Decode: The Architecture Behind Meta’s LLM Serving
• Prefill Is Compute-Bound. Decode Is Memory-Bound. Why Your GPU Shouldn’t Do Both.
• Prefill/decode disaggregation — Ray Serve Docs