An engineering field report on what actually happened when agents went to production — cost equations, governance failures, memory harnesses, and the quiet death of the general-purpose ingestion pipeline.
Motivation
Twelve months ago, the working assumption across most enterprise AI programs was simple: capability scales with model size, so budget for bigger models and wider context windows. That assumption did not survive contact with production. Teams that shipped long-horizon autonomous workflows discovered that the runtime surrounding the agent — context management, routing, governance, and memory — determined success or failure far more than the parameter count of the model inside it.
This report synthesizes what those deployments taught us. It is organized as an experience report: what was tried, what the numbers showed, where things broke, and what we would tell a team starting today. The figures and case studies are drawn from primary sources — Uber's engineering blog, Microsoft's TokenOps publication, Meta's harness and memory research papers, the Reuters investigation into Project OT, and vendor documentation from Cohere, Google Cloud, Trackunit, and Perplexity — all linked in the References section, with verification caveats noted where they apply.
Part 1 — The cost equation that ended "tokenmaxxing"
The first hard lesson was financial. Early agent programs treated AI consumption as an unmanaged cloud expense, and the pattern that emerged — throw a frontier model with an unconstrained context window at every problem — earned the nickname tokenmaxxing. It worked in demos and detonated budgets in production.
The turning point came when engineering teams stopped looking at the monthly invoice as a single number and decomposed total spend into an explicit multiplicative system:
Here U is active users (human and autonomous), S/U is sessions per user, T/S is turns per session, R/T is sub-model requests per turn, K/R is tokens per request, and P/K is unit price per token.
The insight buried in this equation is strategic, not arithmetic. The first two terms — users and session frequency — are growth, and you want them to explode. The last four terms are autonomous overhead: work the agent does on its own behalf, much of it zero-value (redundant retrieval, context bloat, ungrounded execution loops). The entire optimization discipline that followed can be summarized as: let the left side of the equation grow unbounded while engineering the right side toward zero.
Part 2 — Experience report: Uber's software factory
Uber's deployment is the cleanest natural experiment we have, because the team held the underlying model constant while re-engineering everything around it. The starting condition was grim: the annual AI budget was exhausted early in the fiscal cycle. The ending condition, after the runtime rebuild, was aggregate spend flat from April onward while weekly agent requests grew 9.4x and weekly active users grew 7x. With the model unchanged, cost per 1,000 requests fell 34% from peak and cost per session fell 52% from its June high. By the end of the period, more than 70% of code changes were attributed to agents, with over 30,000 skill runs executing daily.
| Metric | Baseline / Peak | Optimized state |
|---|---|---|
| Weekly active agent users | Baseline (Feb) | 7.0x growth |
| Weekly agent requests | Baseline (Feb) | 9.4x growth |
| Aggregate AI spend | Budget exhausted (April) | Flat / stabilized |
| Cost per 1,000 requests | Peak (April) | −34% |
| Cost per session | Peak (June) | −52% |
| Code changes from agents | Minimal | >70% of PRs |
Six runtime mechanisms produced these numbers, and each one is a lesson in where agent spend actually hides.
Benchmark-driven routing. An internal AI gateway routes each subtask based on measured benchmark performance rather than static model assignments. Cheap open-weight and small models absorb simple subtasks; frontier models are reserved for trajectory planning and evaluation. The lesson: most tokens in an agentic workflow do not need frontier intelligence, and a router that knows this empirically beats any hand-assigned model matrix.
Extended prompt cache TTLs. Standard 5-minute prompt cache windows silently expire during human-in-the-loop pauses, because developers think slower than automated subagents. Uber extended interactive session TTL to one hour while keeping 5 minutes for automated subagents, locking in cache-read pricing at roughly 10% of standard input rates. This is the kind of fix that is invisible in an architecture diagram and enormous on an invoice.
Dynamic tool schema extraction. This was the most surprising finding for most teams that replicated it: ingesting hundreds of MCP tool definitions into the context window consumed 50,000–70,000 tokens before the user typed anything. The fix was to route MCP servers through a unified gateway exposing tools as CLI commands that resolve at execution time, so schemas never accumulate in memory.
Code-mode execution batching. The classic agent loop — call a tool, feed the output back into context, reason, call the next tool — multiplies round trips and context growth. Code-mode instead has the model emit a self-contained script that runs the whole multi-step workflow in a local sandbox and returns only a final summary. Reported savings: 50–71% of tokens on basic database queries and over 90% on bulk data tasks. Philosophically, this is the runtime admitting that a deterministic script is a better executor than an LLM narrating its own tool calls.
Context graph grounding. Ungrounded agents explore: they search the web, walk directories, and re-derive facts the organization already knows. Grounding agents against an enterprise context graph — 24 million nodes and 80 million edges spanning 30 internal systems — collapsed one representative query's execution time from 20 minutes to 38 seconds. Exploration is the single most expensive default behavior an agent has.
Terminal-level cost visibility. Rather than hard rationing caps that bottleneck velocity, real-time cost meters were embedded in developer terminal status lines. Making cost ambient rather than punitive changed engineering behavior without changing policy.
Part 3 — When agents share a budget: distributed governance and TokenOps
Single-agent cost control is a solved problem. Multi-agent cost control turned out to be a distributed systems problem, and most teams learned this the expensive way.
The physics are unforgiving: multi-agent systems scale performance by consuming context. Token volume reportedly accounts for 80% of performance variance on complex browsing and research benchmarks, and multi-agent systems consume roughly 15x the tokens of standard chat and 4x the tokens of a single-agent loop. So the architecture that performs best is also the one most likely to blow through a budget cap.
The failure mode is a textbook concurrency bug wearing an AI costume. A planning agent delegates subproblems over HTTP to parallel execution nodes. A global cost cap exists for the run — but each agent tracks spend in its own process memory, starting from zero. Every node individually evaluates itself as within budget while the collective spend sails past the global limit long before any local threshold trips.
Microsoft's TokenOps framework resolves this by shifting governance from the request level to the run level. An entry agent registers a global run ID, propagated to every downstream call via an X-TokenOps-Run-Id HTTP header, binding disparate LLM calls and tool executions across independent services into one accounting context. A shared SQLite ledger (TOKENOPS_DB) replaces isolated local counters as the single source of truth for accumulated spend. In-path SDK hooks intercept each request immediately before execution and run a pre_call_worst_case evaluation — computing the upper-bound cost if the call generated maximum output tokens — and can abort or mutate the call before a single token is committed. When a run approaches its cap, stateful actuators choose between STEER (downgrade the model, trim the context, keep the run alive) and HALT (a deterministic circuit breaker on hard cap breach). Every call is tagged with run ID, agent ID, and step, replacing opaque monthly billing with a granular execution tree of sub-agent cost attribution.
The most instructive design decision is the separation of the decision plane from the execution path. Expensive policy evaluation happens out-of-band; the hot path runs lightweight pre-computed enforcement rules against local memory and reconciles with the central ledger post-call — eventually consistent by default, with an optional strict fail-closed mode for high-cost environments. That trade-off (fast-and-approximately-right versus slow-and-exactly-right) is the same one every distributed rate limiter makes, and recognizing agent governance as that class of problem was the conceptual unlock.
Part 4 — Memory, metacognition, and the decay problem
The second-generation problem, once cost was tamed, was behavioral. Long-horizon agents governed by static system prompts exhibit behavioral state decay: constraints and past failures remain physically present in the context log but progressively lose influence over action selection as the context grows. The instruction is there; the agent just stops acting on it.
Two research directions attacked this from opposite sides.
Learned harnesses (EvoHarness-RL, Meta AI / UIUC). Instead of hand-crafting prompts and state-tracking scaffolds, this framework gives the agent a structured external workspace — Belief, Progress, Experience — and four meta-actions to operate it: track to inspect live environment state into a world-state store, commit to write attempted/pending/blocked subgoals into a progress record, recall to query a cross-episode skill bank of past successful strategies, and note to log failure patterns for a consolidation model to merge into long-term memory at epoch boundaries. Training runs in two phases: supervised fine-tuning to teach the meta-actions exist, then cost-aware reinforcement learning (GRPO) with a reward that explicitly penalizes unnecessary harness calls.
The emergent behavior — harness annealing — is the most interesting result. Early in training the agent queries the workspace at nearly every step. As training progresses, routine patterns internalize into the model's parameters and external calls collapse to near-zero for standard steps, spiking only at genuinely novel bottlenecks. The agent learns when metacognition is worth paying for. The headline number: an 8B open-weight model hit 96.9% on ALFWorld, matching Claude Opus 4.5's reported 96.4% at a fraction of the cost.
Proactive memory agents (Meta). The complementary approach leaves the action agent untouched and runs a second agent beside it, maintaining a structured bank of status, knowledge, and procedural entries. At each turn the memory agent makes one decision: inject a targeted reminder, or stay silent. Attached to Claude Sonnet 4.5, this layer lifted Terminal-Bench 2.0 pass@1 from 37.6% to 45.9% and τ²-Bench from 55.0% to 61.8%. Memory, reframed as an active intervention policy rather than passive vector retrieval, is worth 7–8 points on hard agentic benchmarks.
Part 5 — The failure chapter: Project OT and the Sev 1
No honest experience report skips the disasters, and 2026 supplied two canonical ones, both at Meta.
Project OT was a restructuring initiative to substitute routine human engineering work with autonomous agent workflows. The volume metrics looked like a triumph: overall platform code changes up 220% year over year, per CTO Andrew Bosworth. The value metrics told the real story — feature-delivering changes to end users grew only 36%, major technical and security incidents rose 40%, and engineer time spent remediating disruptions surged 70%. The initiative was abandoned. The postmortem lesson generalizes: ungrounded agents are technical debt factories. Code volume is the easiest metric to inflate and the least correlated with value, and every organization measuring agent success by PR count is currently repeating this experiment.
The Sev 1. An internal engineering agent, operating without mandatory human-in-the-loop review, generated incorrect infrastructure modification instructions and published them directly to an internal developer forum. An engineer executed them. Sensitive user and company data was exposed to unauthorized internal personnel for two hours. The failure chain is worth staring at: the breach required no exploit and no malice — only an agent with publish permissions, a plausible-sounding artifact, and a human who trusted it. The remediation posture that followed is now the baseline recommendation: deterministic policy gateways, explicit permission boundaries, and mandatory human approval hooks on any agent output that can mutate infrastructure or reach an audience.
Part 6 — The specialization horizon
While runtimes matured, the model layer quietly fractured. General-purpose LLMs are being displaced at the ingestion and retrieval layers by narrow, purpose-built infrastructure — and the economics explain why.
Document parsing: Cohere Parse 5. The legacy pipeline — OCR cascaded into a frontier LLM — is expensive and destroys spatial layout. Parse 5 is a 2.3B-parameter vision-language model that consumes pages as images and emits Markdown, HTML tables, or region blocks with bounding boxes. On ParseBench it scores 79.2 overall, with 87.0 on table structure and 86.6 on content faithfulness, while conceding semantic formatting (64.0 — it drops italics and strike-throughs, which for RAG purposes is arguably a feature that saves context). At $1.50 per 1,000 pages against $10.00 for general-purpose hyperscaler parsing, an accounts-payable pipeline processing 13 million pages a month saves roughly $1.47M annually. Throughput is 4.5 pages/second per instance, up to 36 on an 8xH100 node under vLLM. The pattern: trade a few points of frontier accuracy for an order of magnitude in cost and throughput, at exactly the layer where volume is highest.
Multimodal retrieval: Gemini Multimodal Embeddings 2 in Box. Text-only RAG dies quietly inside enterprise repositories, where the decisive context lives in diagrams, flowcharts, and rendered financial tables that flat-text chunking destroys. Google's integration projects text, images, rendered document pages, spreadsheet tables, video (≤128s), and audio (≤180s) into a single 3,072-dimensional vector space, with Matryoshka Representation Learning allowing truncation to 1,536 or 768 dimensions to cut storage cost. Three agent patterns fall out of this: cross-modal search (a text query retrieving a chart from a .pptx), layout-aware page embedding (embedding the rendered page instead of arbitrary text chunks), and cross-document contradiction surfacing (catching a spreadsheet that disagrees with a chart image embedded in a deck).
Vertical telematics: Trackunit IrisX. Industrial AI needs domain integration, not web knowledge. IrisX ingests over 3 billion data points daily from 6M+ connected construction assets and exposes them through an internal MCP server, letting external agents (ChatGPT, Claude, Gemini, Copilot) query machine health, utilization, fault codes, and maintenance schedules in natural language with no custom API work. The under-appreciated component is the "Fit App," a governance tool validating device setup and edge transmission on the physical machine — data-quality enforcement pushed to the point of collection, before bad telemetry can poison the AI layer.
Search economics: Perplexity Sonar. The last specialization lesson is a pricing trap. Developers cost search APIs by token prices and miss the per-request search-context fee:
For a fast factual check (300 tokens in, 100 out on base Sonar at $1.00/M each way), token cost is about $0.0004 — and a medium search-context fee of $0.008 per call means the infrastructure fee is 88–94% of the request cost. Across the SKU ladder (Sonar, Sonar Pro at $3/$15 per M, Sonar Reasoning Pro at $2/$8, Deep Research), search fees of $5–$14 per 1,000 requests dominate high-frequency workloads. Perplexity's response — the Agent API with "Search as Code," unifying iterative search, page fetches, and tool loops under deterministic per-run token and search caps — is the same run-scoped governance pattern from Part 3, arriving at the search layer.
Part 7 — What we would tell a team starting today
Five conclusions survived the year intact.
The runtime is the product. Routing, caching, dynamic schema resolution, and graph grounding moved unit economics and reliability more than any base-model upgrade. Budget your engineering accordingly.
Governance must be run-scoped or it is fiction. Request-level gateways cannot see a distributed multi-agent run. You need context propagation, a shared ledger, worst-case pre-call checks, and a deterministic circuit breaker — or you will rediscover the concurrency bug with your own invoice.
Prompts are being replaced by learned policies. Harness annealing shows that small models can be trained to know when external state is worth querying, matching frontier performance at commodity cost. Static system prompts decay; learned harnesses adapt.
Ingestion and retrieval have gone fully multimodal and specialized. OCR-to-LLM cascades and flat-text chunking are obsolete at enterprise volume. Specialized parsers and unified vector spaces are cheaper and more accurate at the layers that matter.
Autonomy without grounding is negative-value. Project OT and the Sev 1 are the same lesson at two scales: agents that produce unvalidated output faster than humans can review it convert velocity into incidents. Permission boundaries and human approval hooks are not friction — they are the difference between a software factory and a debt factory.
The era of buying intelligence is over. The era of engineering it into bounds has begun.
References
Uber software factory
- Uber Engineering — Running a Software Factory Efficiently at Uber Scale: https://www.uber.com/us/en/blog/efficient-software-factory/
- Axios — Uber cuts AI costs even as usage jumps: https://www.axios.com/2026/08/27/ai-uber-spending
Multi-agent governance
- Microsoft — TokenOps: Real-time, run-scoped cost control for AI agents: https://commandline.microsoft.com/tokenops-real-time-run-scoped-cost-control-ai-agents/
- TokenOps (GitHub): https://github.com/theagentplane/tokenops
- Anthropic — How we built our multi-agent research system: https://www.anthropic.com/engineering/built-multi-agent-research-system
Memory & harnesses
- EvoHarness-RL (Meta AI / UIUC), arXiv:2608.05446: https://arxiv.org/abs/2608.05446
- Remember When It Matters (Meta AI), arXiv:2607.08716: https://arxiv.org/abs/2607.08716
Project OT & Sev 1 incident
- Reuters special report (syndicated) — Zuckerberg's plan to replace Meta staff with AI: https://www.timeslive.co.za/news/sci-tech/2026-08-26-special-report-mark-zuckerberg-had-a-bold-plan-to-replace-meta-staff-with-ai-heres-how-it-imploded/
- TechCrunch — Meta is having trouble with rogue AI agents: https://techcrunch.com/2026/03/18/meta-is-having-trouble-with-rogue-ai-agents
Domain-specialized models
- Cohere — Introducing Parse: https://cohere.com/blog/parse
- Google Cloud — Box + Gemini Multimodal Embeddings 2: https://cloud.google.com/blog/topics/partners/box-ai-agents-gemini-embeddings-multimodal-enterprise-ai
- Trackunit — AI-driven fleet intelligence (IrisX MCP): https://trackunit.com/press/trackunit-introduces-ai-driven-fleet-intelligence/
- Perplexity — API pricing: https://docs.perplexity.ai/docs/getting-started/pricing
