Category: AI Pattern

An engineering field report on what actually happened when agents went to production — cost equations, governance failures, memory harnesses, and the quiet death of the general-purpose ingestion pipeline.

Motivation

Twelve months ago, the working assumption across most enterprise AI programs was simple: capability scales with model size, so budget for bigger models and wider context windows. That assumption did not survive contact with production. Teams that shipped long-horizon autonomous workflows discovered that the runtime surrounding the agent — context management, routing, governance, and memory — determined success or failure far more than the parameter count of the model inside it.

This report synthesizes what those deployments taught us. It is organized as an experience report: what was tried, what the numbers showed, where things broke, and what we would tell a team starting today. The figures and case studies are drawn from primary sources — Uber's engineering blog, Microsoft's TokenOps publication, Meta's harness and memory research papers, the Reuters investigation into Project OT, and vendor documentation from Cohere, Google Cloud, Trackunit, and Perplexity — all linked in the References section, with verification caveats noted where they apply.

Part 1 — The cost equation that ended "tokenmaxxing"

The first hard lesson was financial. Early agent programs treated AI consumption as an unmanaged cloud expense, and the pattern that emerged — throw a frontier model with an unconstrained context window at every problem — earned the nickname tokenmaxxing. It worked in demos and detonated budgets in production.

The turning point came when engineering teams stopped looking at the monthly invoice as a single number and decomposed total spend into an explicit multiplicative system:

Here U is active users (human and autonomous), S/U is sessions per user, T/S is turns per session, R/T is sub-model requests per turn, K/R is tokens per request, and P/K is unit price per token.

The insight buried in this equation is strategic, not arithmetic. The first two terms — users and session frequency — are growth, and you want them to explode. The last four terms are autonomous overhead: work the agent does on its own behalf, much of it zero-value (redundant retrieval, context bloat, ungrounded execution loops). The entire optimization discipline that followed can be summarized as: let the left side of the equation grow unbounded while engineering the right side toward zero.

Part 2 — Experience report: Uber's software factory

Uber's deployment is the cleanest natural experiment we have, because the team held the underlying model constant while re-engineering everything around it. The starting condition was grim: the annual AI budget was exhausted early in the fiscal cycle. The ending condition, after the runtime rebuild, was aggregate spend flat from April onward while weekly agent requests grew 9.4x and weekly active users grew 7x. With the model unchanged, cost per 1,000 requests fell 34% from peak and cost per session fell 52% from its June high. By the end of the period, more than 70% of code changes were attributed to agents, with over 30,000 skill runs executing daily.

Metric	Baseline / Peak	Optimized state
Weekly active agent users	Baseline (Feb)	7.0x growth
Weekly agent requests	Baseline (Feb)	9.4x growth
Aggregate AI spend	Budget exhausted (April)	Flat / stabilized
Cost per 1,000 requests	Peak (April)	−34%
Cost per session	Peak (June)	−52%
Code changes from agents	Minimal	>70% of PRs

Six runtime mechanisms produced these numbers, and each one is a lesson in where agent spend actually hides.

Benchmark-driven routing. An internal AI gateway routes each subtask based on measured benchmark performance rather than static model assignments. Cheap open-weight and small models absorb simple subtasks; frontier models are reserved for trajectory planning and evaluation. The lesson: most tokens in an agentic workflow do not need frontier intelligence, and a router that knows this empirically beats any hand-assigned model matrix.

Extended prompt cache TTLs. Standard 5-minute prompt cache windows silently expire during human-in-the-loop pauses, because developers think slower than automated subagents. Uber extended interactive session TTL to one hour while keeping 5 minutes for automated subagents, locking in cache-read pricing at roughly 10% of standard input rates. This is the kind of fix that is invisible in an architecture diagram and enormous on an invoice.

Dynamic tool schema extraction. This was the most surprising finding for most teams that replicated it: ingesting hundreds of MCP tool definitions into the context window consumed 50,000–70,000 tokens before the user typed anything. The fix was to route MCP servers through a unified gateway exposing tools as CLI commands that resolve at execution time, so schemas never accumulate in memory.

Code-mode execution batching. The classic agent loop — call a tool, feed the output back into context, reason, call the next tool — multiplies round trips and context growth. Code-mode instead has the model emit a self-contained script that runs the whole multi-step workflow in a local sandbox and returns only a final summary. Reported savings: 50–71% of tokens on basic database queries and over 90% on bulk data tasks. Philosophically, this is the runtime admitting that a deterministic script is a better executor than an LLM narrating its own tool calls.

Context graph grounding. Ungrounded agents explore: they search the web, walk directories, and re-derive facts the organization already knows. Grounding agents against an enterprise context graph — 24 million nodes and 80 million edges spanning 30 internal systems — collapsed one representative query's execution time from 20 minutes to 38 seconds. Exploration is the single most expensive default behavior an agent has.

Terminal-level cost visibility. Rather than hard rationing caps that bottleneck velocity, real-time cost meters were embedded in developer terminal status lines. Making cost ambient rather than punitive changed engineering behavior without changing policy.

Part 3 — When agents share a budget: distributed governance and TokenOps

Single-agent cost control is a solved problem. Multi-agent cost control turned out to be a distributed systems problem, and most teams learned this the expensive way.

The physics are unforgiving: multi-agent systems scale performance by consuming context. Token volume reportedly accounts for 80% of performance variance on complex browsing and research benchmarks, and multi-agent systems consume roughly 15x the tokens of standard chat and 4x the tokens of a single-agent loop. So the architecture that performs best is also the one most likely to blow through a budget cap.

The failure mode is a textbook concurrency bug wearing an AI costume. A planning agent delegates subproblems over HTTP to parallel execution nodes. A global cost cap exists for the run — but each agent tracks spend in its own process memory, starting from zero. Every node individually evaluates itself as within budget while the collective spend sails past the global limit long before any local threshold trips.

Microsoft's TokenOps framework resolves this by shifting governance from the request level to the run level. An entry agent registers a global run ID, propagated to every downstream call via an X-TokenOps-Run-Id HTTP header, binding disparate LLM calls and tool executions across independent services into one accounting context. A shared SQLite ledger (TOKENOPS_DB) replaces isolated local counters as the single source of truth for accumulated spend. In-path SDK hooks intercept each request immediately before execution and run a pre_call_worst_case evaluation — computing the upper-bound cost if the call generated maximum output tokens — and can abort or mutate the call before a single token is committed. When a run approaches its cap, stateful actuators choose between STEER (downgrade the model, trim the context, keep the run alive) and HALT (a deterministic circuit breaker on hard cap breach). Every call is tagged with run ID, agent ID, and step, replacing opaque monthly billing with a granular execution tree of sub-agent cost attribution.

The most instructive design decision is the separation of the decision plane from the execution path. Expensive policy evaluation happens out-of-band; the hot path runs lightweight pre-computed enforcement rules against local memory and reconciles with the central ledger post-call — eventually consistent by default, with an optional strict fail-closed mode for high-cost environments. That trade-off (fast-and-approximately-right versus slow-and-exactly-right) is the same one every distributed rate limiter makes, and recognizing agent governance as that class of problem was the conceptual unlock.

Part 4 — Memory, metacognition, and the decay problem

The second-generation problem, once cost was tamed, was behavioral. Long-horizon agents governed by static system prompts exhibit behavioral state decay: constraints and past failures remain physically present in the context log but progressively lose influence over action selection as the context grows. The instruction is there; the agent just stops acting on it.

Two research directions attacked this from opposite sides.

Learned harnesses (EvoHarness-RL, Meta AI / UIUC). Instead of hand-crafting prompts and state-tracking scaffolds, this framework gives the agent a structured external workspace — Belief, Progress, Experience — and four meta-actions to operate it: track to inspect live environment state into a world-state store, commit to write attempted/pending/blocked subgoals into a progress record, recall to query a cross-episode skill bank of past successful strategies, and note to log failure patterns for a consolidation model to merge into long-term memory at epoch boundaries. Training runs in two phases: supervised fine-tuning to teach the meta-actions exist, then cost-aware reinforcement learning (GRPO) with a reward that explicitly penalizes unnecessary harness calls.

The emergent behavior — harness annealing — is the most interesting result. Early in training the agent queries the workspace at nearly every step. As training progresses, routine patterns internalize into the model's parameters and external calls collapse to near-zero for standard steps, spiking only at genuinely novel bottlenecks. The agent learns when metacognition is worth paying for. The headline number: an 8B open-weight model hit 96.9% on ALFWorld, matching Claude Opus 4.5's reported 96.4% at a fraction of the cost.

Proactive memory agents (Meta). The complementary approach leaves the action agent untouched and runs a second agent beside it, maintaining a structured bank of status, knowledge, and procedural entries. At each turn the memory agent makes one decision: inject a targeted reminder, or stay silent. Attached to Claude Sonnet 4.5, this layer lifted Terminal-Bench 2.0 pass@1 from 37.6% to 45.9% and τ²-Bench from 55.0% to 61.8%. Memory, reframed as an active intervention policy rather than passive vector retrieval, is worth 7–8 points on hard agentic benchmarks.

Part 5 — The failure chapter: Project OT and the Sev 1

No honest experience report skips the disasters, and 2026 supplied two canonical ones, both at Meta.

Project OT was a restructuring initiative to substitute routine human engineering work with autonomous agent workflows. The volume metrics looked like a triumph: overall platform code changes up 220% year over year, per CTO Andrew Bosworth. The value metrics told the real story — feature-delivering changes to end users grew only 36%, major technical and security incidents rose 40%, and engineer time spent remediating disruptions surged 70%. The initiative was abandoned. The postmortem lesson generalizes: ungrounded agents are technical debt factories. Code volume is the easiest metric to inflate and the least correlated with value, and every organization measuring agent success by PR count is currently repeating this experiment.

The Sev 1. An internal engineering agent, operating without mandatory human-in-the-loop review, generated incorrect infrastructure modification instructions and published them directly to an internal developer forum. An engineer executed them. Sensitive user and company data was exposed to unauthorized internal personnel for two hours. The failure chain is worth staring at: the breach required no exploit and no malice — only an agent with publish permissions, a plausible-sounding artifact, and a human who trusted it. The remediation posture that followed is now the baseline recommendation: deterministic policy gateways, explicit permission boundaries, and mandatory human approval hooks on any agent output that can mutate infrastructure or reach an audience.

Part 6 — The specialization horizon

While runtimes matured, the model layer quietly fractured. General-purpose LLMs are being displaced at the ingestion and retrieval layers by narrow, purpose-built infrastructure — and the economics explain why.

Document parsing: Cohere Parse 5. The legacy pipeline — OCR cascaded into a frontier LLM — is expensive and destroys spatial layout. Parse 5 is a 2.3B-parameter vision-language model that consumes pages as images and emits Markdown, HTML tables, or region blocks with bounding boxes. On ParseBench it scores 79.2 overall, with 87.0 on table structure and 86.6 on content faithfulness, while conceding semantic formatting (64.0 — it drops italics and strike-throughs, which for RAG purposes is arguably a feature that saves context). At $1.50 per 1,000 pages against $10.00 for general-purpose hyperscaler parsing, an accounts-payable pipeline processing 13 million pages a month saves roughly $1.47M annually. Throughput is 4.5 pages/second per instance, up to 36 on an 8xH100 node under vLLM. The pattern: trade a few points of frontier accuracy for an order of magnitude in cost and throughput, at exactly the layer where volume is highest.

Multimodal retrieval: Gemini Multimodal Embeddings 2 in Box. Text-only RAG dies quietly inside enterprise repositories, where the decisive context lives in diagrams, flowcharts, and rendered financial tables that flat-text chunking destroys. Google's integration projects text, images, rendered document pages, spreadsheet tables, video (≤128s), and audio (≤180s) into a single 3,072-dimensional vector space, with Matryoshka Representation Learning allowing truncation to 1,536 or 768 dimensions to cut storage cost. Three agent patterns fall out of this: cross-modal search (a text query retrieving a chart from a .pptx), layout-aware page embedding (embedding the rendered page instead of arbitrary text chunks), and cross-document contradiction surfacing (catching a spreadsheet that disagrees with a chart image embedded in a deck).

Vertical telematics: Trackunit IrisX. Industrial AI needs domain integration, not web knowledge. IrisX ingests over 3 billion data points daily from 6M+ connected construction assets and exposes them through an internal MCP server, letting external agents (ChatGPT, Claude, Gemini, Copilot) query machine health, utilization, fault codes, and maintenance schedules in natural language with no custom API work. The under-appreciated component is the "Fit App," a governance tool validating device setup and edge transmission on the physical machine — data-quality enforcement pushed to the point of collection, before bad telemetry can poison the AI layer.

Search economics: Perplexity Sonar. The last specialization lesson is a pricing trap. Developers cost search APIs by token prices and miss the per-request search-context fee:

For a fast factual check (300 tokens in, 100 out on base Sonar at $1.00/M each way), token cost is about $0.0004 — and a medium search-context fee of $0.008 per call means the infrastructure fee is 88–94% of the request cost. Across the SKU ladder (Sonar, Sonar Pro at $3/$15 per M, Sonar Reasoning Pro at $2/$8, Deep Research), search fees of $5–$14 per 1,000 requests dominate high-frequency workloads. Perplexity's response — the Agent API with "Search as Code," unifying iterative search, page fetches, and tool loops under deterministic per-run token and search caps — is the same run-scoped governance pattern from Part 3, arriving at the search layer.

Part 7 — What we would tell a team starting today

Five conclusions survived the year intact.

The runtime is the product. Routing, caching, dynamic schema resolution, and graph grounding moved unit economics and reliability more than any base-model upgrade. Budget your engineering accordingly.

Governance must be run-scoped or it is fiction. Request-level gateways cannot see a distributed multi-agent run. You need context propagation, a shared ledger, worst-case pre-call checks, and a deterministic circuit breaker — or you will rediscover the concurrency bug with your own invoice.

Prompts are being replaced by learned policies. Harness annealing shows that small models can be trained to know when external state is worth querying, matching frontier performance at commodity cost. Static system prompts decay; learned harnesses adapt.

Ingestion and retrieval have gone fully multimodal and specialized. OCR-to-LLM cascades and flat-text chunking are obsolete at enterprise volume. Specialized parsers and unified vector spaces are cheaper and more accurate at the layers that matter.

Autonomy without grounding is negative-value. Project OT and the Sev 1 are the same lesson at two scales: agents that produce unvalidated output faster than humans can review it convert velocity into incidents. Permission boundaries and human approval hooks are not friction — they are the difference between a software factory and a debt factory.

The era of buying intelligence is over. The era of engineering it into bounds has begun.

References

Uber software factory

1. Uber Engineering — Running a Software Factory Efficiently at Uber Scale: https://www.uber.com/us/en/blog/efficient-software-factory/
2. Axios — Uber cuts AI costs even as usage jumps: https://www.axios.com/2026/08/27/ai-uber-spending

Multi-agent governance

3. Microsoft — TokenOps: Real-time, run-scoped cost control for AI agents: https://commandline.microsoft.com/tokenops-real-time-run-scoped-cost-control-ai-agents/
4. TokenOps (GitHub): https://github.com/theagentplane/tokenops
5. Anthropic — How we built our multi-agent research system: https://www.anthropic.com/engineering/built-multi-agent-research-system

Memory & harnesses

6. EvoHarness-RL (Meta AI / UIUC), arXiv:2608.05446: https://arxiv.org/abs/2608.05446
7. Remember When It Matters (Meta AI), arXiv:2607.08716: https://arxiv.org/abs/2607.08716

Project OT & Sev 1 incident

8. Reuters special report (syndicated) — Zuckerberg's plan to replace Meta staff with AI: https://www.timeslive.co.za/news/sci-tech/2026-08-26-special-report-mark-zuckerberg-had-a-bold-plan-to-replace-meta-staff-with-ai-heres-how-it-imploded/
9. TechCrunch — Meta is having trouble with rogue AI agents: https://techcrunch.com/2026/03/18/meta-is-having-trouble-with-rogue-ai-agents

Domain-specialized models

10. Cohere — Introducing Parse: https://cohere.com/blog/parse
11. Google Cloud — Box + Gemini Multimodal Embeddings 2: https://cloud.google.com/blog/topics/partners/box-ai-agents-gemini-embeddings-multimodal-enterprise-ai
12. Trackunit — AI-driven fleet intelligence (IrisX MCP): https://trackunit.com/press/trackunit-introduces-ai-driven-fleet-intelligence/
13. Perplexity — API pricing: https://docs.perplexity.ai/docs/getting-started/pricing

Procedural Memory, Failure Dynamics, Retrieval Scaling, and the Path to Autonomous Lifelong Learning Systems

Executive Summary

Skills — modular, reusable procedural instructions injected into an agent's context — have become a cornerstone of modern agentic AI systems. Yet the industry has largely built skill libraries through intuition and trial and error, leaving fundamental questions unanswered: When do skills help, why do they work, and where do they fail?

Demystifying Agent Skills: Why They Work—Until They Don't (Jiang et al., 2026) [1] — a controlled evaluation of 8,135 trial records — combined with related literature on skill-induced failures [2], embodied code memory [3], marketplace security [4], and multi-agent compilation [5], provides the answers. The central findings:

1. Skills are anchors, not encyclopedias. Procedural anchoring — stabilizing execution sequences — accounts for 65.7% of successful skill cases; explicit factual knowledge injection accounts for only 4.5%.
2. Format matters as much as experience. Distilling raw trajectories into standardized skill files improves task success by +6.06 percentage points over injecting the same experience as raw workflow logs.
3. Unannotated experience is toxic. Withholding success/failure labels during skill distillation collapses downstream success from 74.6% to 40.0%.
4. Retrieval precision collapses at scale, but task success doesn't. Growing a catalog from 5 to 100 skills drops exact-retrieval precision from 29.6% to 3.3%, yet downstream success holds steady at ~36–39% — because related "distractor" skills still provide partial procedural support.
5. Skills trade one failure class for another. Infrastructure setup faults nearly vanish (5.3% → 0.2%, a 96% reduction), but rigid runbook-following raises invocation/boundary faults (3.6% → 14.8%).

The report closes with architectural guidance: two-level retrieval gating, MAS-to-SAS compilation, FCoT 3.0 governance, and a six-level agent maturity framework culminating in autonomous lifelong learning systems.

1. Methodology: The Tri-Mode Evaluation Paradigm

To scientifically isolate the effect of skills, researchers ran controlled experiments on multi-step environments that require complex execution, debugging, and verification. Agents were evaluated across three execution modes using identical prior experience:

              ┌─────────────────────────────────────────┐
              │        Raw Execution Baseline           │
              │  (Zero prior context / No skill files)  │
              └────────────────────┬────────────────────┘
                                   │
          ┌────────────────────────┴────────────────────────┐
          ▼                                                 ▼
┌───────────────────────┐                     ┌───────────────────────┐
│   Workflow Memory     │                     │    Distilled Skill    │
│ (Uncompressed logs &  │                     │  (Standardized        │
│   command history)    │                     │   SKILL.md runbooks)  │
└───────────────────────┘                     └───────────────────────┘

1. Raw Execution — The agent tackles the task with no historical context or skill files, relying strictly on pre-trained knowledge.
2. Workflow Memory — The agent receives uncompressed, raw historical execution logs and command traces injected into its prompt context.
3. Distilled Skill — The agent receives a clean, standardized SKILL.md file containing curated operational checklists extracted from those same past runs.

Benchmark Evaluation Matrix

Benchmark Suite	Total Trials	Evaluation Focus
Terminal-Bench 2.0	4,210	Software engineering, CLI automation, build/deploy scripts
SkillsBench	3,925	Cross-domain tool usage, API integrations, data transformations

Tri-Mode Performance Results

Execution Mode	Infrastructure Setup Faults	Downstream Task Success
Raw Execution	5.3%	50.0% (baseline)
Workflow Memory	3.8%	55.7%
Distilled Skill	0.2% (−96%)	61.7% (+6.06 pts vs. memory)

Key methodological takeaway: Distilling experience into a standardized skill improves task success over raw workflow memory by 6.06 percentage points — proving that how experience is formatted matters as much as possessing the experience in the first place. Developers can replicate this baseline test locally to verify whether their custom abstractions actually add value over simply injecting raw logs into the prompt.

2. Functional Mechanics: Runbooks vs. Tutorials

A widespread misconception among AI engineers is treating skills as "tutorials" or "textbooks" intended to teach the agent facts or general algorithms. The empirical data refutes this decisively. Skills do not act as repositories for missing knowledge; they act as anchors that prevent agents from getting derailed during complex workflows.

Breakdown of Skill Utility Determinants

Skill Utility = f(Anchoring, Warnings, Checklists, Adaptation, Facts)

Functional Mechanism	Relative Contribution	Operational Role
Procedural Anchoring	65.7%	Enforces deterministic command sequencing, flag syntax, and execution order
Pitfall Avoidance Warnings	11.2%	Prevents known environment anti-patterns and deprecation traps
Runtime Validation Checklists	8.6%	Mandates pre-flight verification, health probes, and output checks
Contextual Adaptation Guidance	6.0%	Guides dynamic path, port, and environment variable substitutions
Explicit Knowledge Injection	4.5%	Direct factual or domain-specific instruction
Residual Execution Utility	4.0%	Miscellaneous structural formatting benefits

The Runbook Principle

Skills succeed because they provide deterministic operational checklists. Rather than fixing high-level reasoning, they deliver execution robustness — preventing the model from entering exploratory trial-and-error loops. The clearest evidence: environment infrastructure failures (tooling configuration, dependency workarounds, general environment setup) drop from 5.3% in raw execution to 0.2% with distilled skills.

Case Study: React Patch Latency Optimization (Terminal-Bench 2.0)

• Problem: An agent was tasked with patching a React application to achieve a warm page-load latency below 800 ms.
• Raw Execution: The agent correctly identified the bottleneck and wrote a functionally correct patch — but failed the automated verification check. The API took 915 ms to load because the agent left independent requests serialized.
• Workflow Memory: Injected with raw past logs, the agent replayed stale setup commands and unnecessary build configuration steps, wasting cycles without addressing the concurrency issue.
• Distilled Skill: The skill acted as a strict procedural runbook:

1. Audit component hooks and convert independent await calls to Promise.all().
2. Initiate promise execution early in the component lifecycle and await resolution late.
3. Run local bundle latency profiling (e.g., webpack-bundle-analyzer) before committing.

• Result: Anchored by the checklist, the agent passed all latency and functional checks in 11.74 seconds.

>

> Takeaway for developers: Stop writing skills to teach your agents facts or generic algorithms. Format your skills as standardized, step-by-step runbooks that force the agent to follow a strict operational checklist.

3. Trajectory Quality: The Hazard of Unannotated Experience

Automatically generating skills from continuous execution logging is a popular design pattern — and a dangerous one. Skills "fail under brittle assumptions, incompatible contexts, or insufficient adaptation," and a major cause traces back to how the skills were created: distilling from raw trajectories without telling the compiler which runs succeeded and which failed severely degrades performance.

The "No-Hint" Ablation Experiment

Researchers gave a skill compiler a mixed historical batch of 3 successful and 2 failed trajectories (3s2f) under two conditions:

• Annotated: trajectory files carried explicit outcome labels ([SUCCESS] / [FAILURE]).
• Unannotated ("No-Hint"): outcome labels were withheld, forcing the compiler to treat all runs as equally valid experience.

Impact of Trajectory Outcome Annotations on Task Success

┌────────────────────────────────────────────────────────────────┐
│ Annotated Trajectories (3s2f)      ████████████████████ 74.6%  │
│ Unannotated Trajectories (No-Hint) ██████████           40.0%  │
│                                             (−34.6 pt plunge)  │
└────────────────────────────────────────────────────────────────┘

Without explicit outcome hints, the compiler could not separate signal from noise — treating exploratory workarounds, failed shell attempts, and hallucinated commands as canonical procedure, permanently baking anti-patterns into the skill library.

Optimal Trajectory Batch Composition

Varying the mix of successful (s) and failed (f) trajectories reveals a balanced ratio outperforms success-only distillation:

Trajectory Ratio	Annotation	Downstream Success	Behavioral Dynamic
5s / 0f	Annotated	68.4%	Positive mimicry; lacks negative edge-case awareness
3s / 2f	Annotated	78.1%	Optimal: learns positive paths and extracts explicit defensive rules
3s / 2f	Unannotated	40.0%	Catastrophic: anti-patterns baked in permanently
0s / 5f	Annotated	47.7%	Defensive rules only; no positive execution templates

> Engineering directive: Never feed raw, unannotated terminal logs into a skill compiler. Use an LLM-as-a-Judge (or outcome oracle) to label every trajectory before distillation, and prefer a balanced success/failure mix so the resulting skill encodes both what to do and what to avoid.

4. Failure Mode Taxonomy & Boundary Risks

Open-coding analysis of the benchmark transcripts identifies three primary outcome categories:

• SC1 — Task Success: Clean completion satisfying all verification criteria.
• SC2 — Execution & Setup Faults: Infrastructure configuration failures, missing dependencies, shell syntax errors, broken tool invocations.
• SC3 — Invocation & Boundary Faults: Rigid skill misapplication, incorrect parameter formatting, false precondition assumptions, unrecoverable procedural loops.

Failure Mode Distribution Across Execution Modes

Category	Raw	Memory	Skill
SC1: Task Success	50.0%	55.7%	61.7%
SC2: Execution & Setup	37.3%	31.8%	23.5%
— Infrastructure setup subtype	(5.3%)	(1.8%)	(0.2%)
SC3: Invocation / Boundary	3.6%	8.5%	14.8%

The Trade-Off: Setup Fault Collapse vs. Boundary Fault Spike

Skills nearly eliminate low-level environment configuration failures (a 96% reduction in setup faults). But they introduce a new hazard: SC3 boundary faults quadruple (3.6% → 14.8%). When agents rely on static runbooks, they become vulnerable to mechanical misapplication — rigidly executing checklist steps even when local edge conditions have invalidated the runbook's assumptions, triggering unrecoverable rigid loops.

This is the core lifecycle tension of skill engineering: procedural discipline stabilizes routine execution but degrades adaptive judgment at the boundaries. Mitigations (boundary guardrails, fallback exits, thinking-budget caps) are covered in Sections 9–10.

5. Scaling Mechanics: Retrieval Collapse & Two-Level Gating

The Retrieval Collapse Paradox

As a skill pool grows from k = 5 to k = 100 options under flat semantic embedding search, the actual-use precision of retrieving the exact ground-truth skill falls drastically — yet downstream task success remains stable:

Catalog Size (k)	Exact Retrieval Precision	Downstream Task Success
5	29.6%	36.4%
10	15.2%	37.3%
20	9.4%	37.6%
50	4.8%	38.0%
100	3.3% (collapse)	39.3% (sustained)

Why Task Success Holds Flat: Partial Procedural Support

The data reveals that exact ground-truth invocation is neither sufficient nor necessary for success. When retrieval selects a "distractor" skill from the same general category — e.g., a general API-debugging skill instead of the exact checkout-API-debugging skill — the agent still receives a valuable procedural checklist that guides execution well enough to pass.

The real danger in large catalogs is semantic confusability: when a pool fills with skills that look nearly identical in embedding space, offline retrieval systems cannot reliably discriminate between them.

Overcoming Semantic Confusability: Two-Level Gating Architecture

Production systems should replace flat vector search with a two-level gate:

Incoming User Task / System Exception

                        │
                        ▼
┌──────────────────────────────────────────────────────────────┐
│  LEVEL 1: Domain Bucket Router (LLM Meta-Classifier)         │
│  Narrows context to a high-level domain (frontend, ops, db)  │
└───────────────────────────────┬──────────────────────────────┘
                                │
                                ▼
┌──────────────────────────────────────────────────────────────┐
│  LEVEL 2: Strict Trigger Rules & Regex                       │
│  Skill fires only on exact conditions, e.g. stdout matches   │
│  "Connection refused"                                        │
└───────────────────────────────┬──────────────────────────────┘
                                │
                                ▼

Target SKILL.md Loaded into Prompt Context

1. Level 1 — Domain Bucket Routing: A lightweight LLM meta-router categorizes the incoming task into a broad bucket (e.g., frontend-build, database, ops), collapsing the candidate set before any similarity search occurs.
2. Level 2 — Strict Trigger Conditions: Within the isolated bucket, a skill is injected only when a rigid, programmatic rule is met — such as detecting an exact error string in terminal logs.

>

> Takeaway for developers: If your catalog exceeds ~20 skills, flat vector retrieval is already degrading. Gate by domain buckets first, strict triggers second.

6. Context Economics & Token Consumption

Skills fundamentally shift context-window economics, trading increased input prompt tokens for reduced exploratory output generation:

Execution Mode	Mean Input Tokens	Mean Output Tokens	Total Burden
Raw Execution	541.5K	14.2K	555.7K
Workflow Memory	417.9K	8.3K	426.2K
Skill Injection	511.7K	9.8K	521.5K

• Raw execution burns tokens on unguided trial-and-error loops, generating high volumes of exploratory output.
• Workflow memory reduces output volume but injects stale process noise and irrelevant intermediate states that risk derailing execution.
• Skill injection balances the trade: structured guidance up front cuts wasteful exploration while avoiding log noise, and standardized skill prefixes improve KV-cache reuse.

7. Cross-Domain Literature Synthesis

Related agent-skills publications reinforce and extend the core empirical findings.

Comparative Matrix

Study	Primary Focus	Skill Representation	Key Finding
Demystifying Agent Skills — Jiang et al. (2026) [1]	8,135-trial controlled evaluation	SKILL.md markdown runbooks	65.7% procedural anchoring vs. 4.5% knowledge injection; +6.06 pt distillation gain; retrieval collapse 29.6% → 3.3%
Agent Skills Can Be Harmful — Dong et al. (2026) [2]	Attribution of 307 skill-induced failures (125 functional + 182 efficiency)	Skills loaded into SkillsBench & SWE-Skills-Bench runs	Task-Implementation Faults = 68.8% of functional failures; Excessive Procedure = 62.6% of efficiency regressions; SkillTriage tool
Voyager — Wang et al. (2023) [3]	Embodied agent memory	Executable JavaScript code blocks	Executable skills in a vector DB with automatic curriculum learning
Credential Leakage in LLM Agent Skills + SkillProbe (2026) [4]	Marketplace security (170,226 SkillsMP skills; 17,022 sampled)	Marketplace skill packages	520 affected skills, 1,708 issues, 10-pattern leakage taxonomy; 76.3% of cases detectable only cross-modally
When Single-Agent with Skills Replace Multi-Agent Systems — Li (2026) [5]	MAS-to-SAS compilation & skill-selection scaling	Compiled skill libraries as internalized agent behaviors	Compilation cuts tokens/latency at competitive accuracy; selection degrades as a phase transition under semantic confusability

7.1 When Skills Actively Harm: Failure Attribution (Dong et al., Agent Skills Can Be Harmful)

Dong et al. built a differential analysis framework that attributes a failure or cost regression to a specific loaded skill by comparing the skill-guided run against a no-skill (or semantically matched) reference run on the same task. Instantiated on SkillsBench and SWE-Skills-Bench, this yields 307 skill-induced failures — 125 functional failures and 182 efficiency regressions:

Functional failures (125 cases):

• Task-Implementation Faults (86/125 = 68.8%): Seemingly relevant skills — not obviously irrelevant ones — cause agents to incorrectly implement or omit task-required elements, rigidly following runbook guidance despite conditions that invalidate it.
• Wrong Artifact Locations (24 cases): Outputs generated in temporary directories unaligned with the evaluation harness.
• Environment Mismatch (13 cases): Dependency-version conflicts between runbook assumptions and the local container.

Efficiency regressions (182 cases): These are not explained by prompt length alone. Where regressions do come from context overhead, mandatory skill-body text accounts for nearly all of it (43 of 46 cases). But the dominant category is Excessive Procedure (114/182 = 62.6%) — chiefly excessive verification (67 cases) and heavy implementation pipelines (30 cases). In other words, skills often turn validation checklists into cost sinks — the same runtime-validation mechanism that contributes 8.6% of skill utility (Section 2) becomes the largest source of skill-induced waste when over-specified.

Dong et al. also ship SkillTriage, a taxonomy-guided attribution tool that normalizes paired cases, extracts differential evidence, and produces triage reports — essential infrastructure for deciding whether to fix the skill, the retrieval, or the base prompt.

7.2 Executable Code Memory & Curriculum (Voyager)

Voyager demonstrated that in embodied and code-execution environments (Minecraft JavaScript execution), skills are best stored as modular, executable functions in a vector database, accumulated through an automatic curriculum. This complements the runbook paradigm: markdown runbooks anchor procedure, while executable code skills encode verified capability.

7.3 Marketplace Security & Credential Leakage

The first large-scale empirical study of credential leakage in agent skills started from 170,226 skills on SkillsMP (the largest open-source skill marketplace), sampled 17,022 via stratified random sampling, and subjected each to static analysis (regex and AST-based secret extraction), dynamic sandbox testing with mock credentials, and manual cross-referencing of developer intent against runtime behavior. It identified 520 affected skills harboring 1,708 security issues and derived a taxonomy of 10 leakage patterns — 4 arising from developer negligence, 6 from deliberate adversarial construction.

The pivotal finding: credential leakage is fundamentally cross-modal — 76.3% of cases surface only when natural-language skill descriptions and executable code are analyzed jointly. Static scanning of either layer alone misses most of the risk. The companion SkillProbe framework extends this with multi-agent security auditing, exposing the semantic gap between "safe"-sounding documentation and shadow capabilities (unauthorized access, hidden exfiltration) in the underlying code — and showing that skills benign in isolation can combine into lethal cross-skill attack chains. An ecosystem-scale companion audit of 31,132 skills found 26.1% contained at least one vulnerability, with script-bearing skills roughly 2.1× more likely to be vulnerable.

Implications for skill pipelines: any distillation pipeline must include automated secret scrubbing (regex + AST + entropy-based) on stdout/stderr before persistence, plus cross-modal auditing that checks whether the skill's stated intent matches its executable behavior — the exact controls formalized in Sublevel 6a of the maturity model.

8. Architectural Paradigm: Multi-Agent Systems (MAS) vs. Skill-Augmented Single Agents (SAS)

A critical enterprise design decision is choosing between multi-agent graph topologies and a single agent with a dynamic skill library.

Multi-Agent System (MAS) Single-Agent with Skills (SAS)

  ┌────────────┐  Msg  ┌──────────┐       ┌────────────────────────────┐
  │ Coordinator├──────▶│ Engineer │       │       Single Agent         │
  └─────┬──────┘       └────┬─────┘       │  ┌──────────────────────┐  │
        │ Msg               │ Msg         │  │   Active SKILL.md    │  │
        ▼                   ▼             │  │ (Compiled Role Rules)│  │
  ┌────────────┐      ┌──────────┐        │  └──────────────────────┘  │
  │  Reviewer  │      │  Tester  │        └────────────────────────────┘
  └────────────┘      └──────────┘

Architectural Comparison Matrix

Dimension	Multi-Agent System (MAS)	Single Agent + Skills (SAS)
Token overhead	High (multiplied inter-agent message passing)	Low (−54% prompt tokens)
Execution latency	High (sequential multi-turn handoffs)	Low (−50% execution time)
Role context isolation	Complete (distinct context windows)	Bounded by prompt window
Catalog scaling	High graph modularity	Degrades at k ≥ 20 without gating
Model heterogeneity	Native (mix SLMs + frontier LLMs)	Homogeneous base model
KV-cache reuse	Low across handoffs	High (shared prefix caching)
Error debuggability	Cascading conversational drift	Isolated runbook execution log

MAS-to-SAS Compilation

Li (2026, When Single-Agent with Skills Replace Multi-Agent Systems and When They Fail) formalizes this by viewing skills as internalized agent behaviors: a multi-agent system can be compiled into an equivalent single-agent system, trading inter-agent communication for skill selection. Each specialized agent's function is distilled into one or more skills via a compilation mapping Φ:

Φ : M = ⟨A, G, Π⟩ → S = ⟨a, S_Φ, σ_Φ⟩

Preliminary experiments show this substantially reduces token usage and latency while maintaining competitive accuracy on reasoning benchmarks (the −54%/−50% figures in the matrix above reflect the synthesis material's estimates of this effect).

The critical caveat — "and when they fail": Li's deeper contribution is the scaling limit. Skill selection exhibits bounded capacity analogous to human decision-making: selection accuracy degrades not gradually but as a phase transition once the library crosses a threshold, driven by semantic confusability among similar skills. Hierarchical routing mitigates the overload — independently corroborating the retrieval-collapse data and two-level gating architecture of Section 5. You cannot simply keep adding skills; past the threshold, selection breaks down suddenly.

Decision heuristic: Use MAS where role isolation and model heterogeneity are essential; compile to SAS where token cost, latency, and cache efficiency dominate — but only with hierarchical/gated retrieval in place, since compilation converts a coordination problem into a selection problem that has its own failure regime.

9. Fractal Chain-of-Thought (FCoT 3.0) & the 5R Governance Protocol

The SC3 boundary-fault spike (Section 4) shows that skills can trap agents in rigid, unrecoverable execution loops. To prevent this, skill execution should be embedded inside a recursive, self-verifying reasoning substrate — Fractal Chain-of-Thought 3.0 — governed by the 5R protocol:

              ┌────────────────────────┐
              │      1. REFLECT        │
              │  Verify Preconditions  │
              └───────────┬────────────┘
                          ▼
              ┌────────────────────────┐
              │      2. REASON         │
              │ Structured Logic Tree  │
              └───────────┬────────────┘
                          ▼
┌────────────────┐  ┌──────────────────────┐  ┌────────────────┐
│   3. RETRY     │◄─┤ 4. RESUBSTANTIATE    ├─►│   5. REPORT    │
│ Local scoped   │  │ Validate output vs.  │  │ Status & token │
│ retries        │  │ environment state    │  │ telemetry      │
└────────────────┘  └──────────────────────┘  └────────────────┘

1. Reflect — Audit skill assumptions, preconditions, and required tool availability before executing any command.
2. Reason — Maintain structured, verifiable step-by-step logic paths throughout tool interaction sequences.
3. Retry — Execute logic-aware retries within local sub-trees, without polluting the parent conversation context.
4. Resubstantiate — Validate execution outputs against ground-truth environment checks (port availability, file existence, test passage) before committing final state.
5. Report — Stream real-time execution status, thinking-budget telemetry, and error metrics to supervisor nodes.

The recursive, multi-aperture structure lets macro-planning and micro-execution operate at different granularities while the verification gates (Resubstantiate) and budget telemetry (Report) provide the anti-rigidity escape hatches that static runbooks lack.

10. The Agent Maturity Hierarchy

Progression from primitive prompting to self-evolving architectures can be mapped against the empirical failure data:

┌──────────────────────────────────────────────────────────────────┐
│  LEVEL 6: Autonomous Lifelong Learning Systems                   │
│    ├── 6a: Trajectory Annotation & Security Curation             │
│    ├── 6b: Two-Level Gating & Anti-Rigidity Guardrails           │
│    └── 6c: Closed-Loop FCoT & Dynamic MAS Compilation            │
├──────────────────────────────────────────────────────────────────┤
│  LEVEL 5: Multi-Agent Swarms & Graph Topologies                  │
├──────────────────────────────────────────────────────────────────┤
│  LEVEL 4: Skill-Augmented Single Agents (SAS)                    │
├──────────────────────────────────────────────────────────────────┤
│  LEVEL 3: Workflow Trace Memory Systems                          │
├──────────────────────────────────────────────────────────────────┤
│  LEVEL 2: Tool-Reactive ReAct Loops                              │
├──────────────────────────────────────────────────────────────────┤
│  LEVEL 1: Conversational Prompting Baseline                      │
└──────────────────────────────────────────────────────────────────┘

Foundational Levels (1–5)

Level 1 — Conversational Prompting. Zero-shot / few-shot prompts with no persistent state, tool interfaces, or external memory. Characteristic failures: context saturation and the absence of verifiable deterministic ground truth.

Level 2 — Tool-Reactive Loops. Basic ReAct-style execution with function calling against unanchored API schemas. Characteristic failures: the 5.3% setup-error rate driven by argument hallucination, parameter drift, and malformed tool calls.

Level 3 — Workflow Trace Memory. Raw, uncompressed execution logs injected directly into prompt context. Characteristic failures: process-noise amplification, irrelevant intermediate states, and token bloat that degrade rather than aid performance.

Level 4 — Skill-Augmented Single Agents (SAS). Static SKILL.md runbook libraries encapsulating environmental pre-flight checks, deterministic execution templates, and scoped APIs. Delivers the 96% setup-fault reduction — but is prone to over-proceduralization: SC3 boundary failures when edge conditions trigger rigid, unrecoverable loops.

Level 5 — Multi-Agent Swarms. Graph topologies (hierarchical, peer-to-peer, routing-hub) with specialized agent personas and isolated context windows. Characteristic failures: inter-agent communication latency, token explosion across message buses, and synchronization bottlenecks.

Level 6 — Autonomous Lifelong Learning Systems

Level 6 represents self-evolving platforms that continuously refine their own skill catalogs without human intervention, decomposed into three sublevels — each directly addressing an empirically identified failure mode:

Sublevel 6a — Trajectory Annotation, Environment Synthesis & Security Curation

Addresses: the No-Hint hazard and marketplace credential leaks.

• LLM Outcome Judge: Automated trajectory labeling enforcing the optimal 3s2f distillation ratio (78.1% downstream success), preventing the 40.0% collapse caused by unannotated logs.
• AST Security Auditing: Static analysis of dynamically authored and third-party skill scripts to catch injection and unsafe patterns before they enter the library.
• Secret Sanitization & Cross-Modal Auditing: Automated regex-, AST-, and entropy-based scrubbing of credentials, JWTs, and API keys from stdout/stderr before any log content persists into skill memory — paired with cross-modal checks that a skill's stated intent matches its executable behavior, since 76.3% of real-world leakage cases are detectable only by analyzing description and code jointly.

Sublevel 6b — Dynamic Retrieval & Anti-Rigidity Guardrails

Addresses: retrieval collapse and SC3 boundary faults.

• Two-Level Gating Engine: Domain-bucket LLM routers paired with strict regex/error-log triggers, eliminating the 29.6% → 3.3% precision collapse at scale. Multi-stage pipelines (sparse schema match → dense rerank → selection-agent filter) provide sublinear index scaling.
• Boundary Fallback Monitoring: Automated detection of divergence between runbook steps and observed environment state, with explicit fallback exits instead of persistent looping on anomalies.
• Thinking-Budget Caps: Hard limits on recursive reasoning tokens to prevent infinite verification loops.

Sublevel 6c — Closed-Loop FCoT & Dynamic MAS Compilation

Addresses: architectural cost and continuous self-improvement.

• FCoT 3.0 Execution Substrate: All skill execution runs inside tree-structured reasoning nodes governed by the 5R protocol (Reflect, Reason, Retry, Resubstantiate, Report).
• Differential Skill Triage: Continuous auditing that distinguishes transient environmental failures from genuine skill defects from base-model faults, routing each to the appropriate fix.
• Dynamic MAS-to-SAS Compilation: Runtime evaluation of multi-agent graphs, extraction of the critical deterministic path, and compilation into optimized single-agent skill libraries — capturing the −54% token / −50% latency gains while preserving specialization.

11. Strategic Engineering Directives

1. Format skills as operational runbooks, not tutorials. Write step-by-step deterministic checklists with exact commands, ordering constraints, and verification gates. Procedural anchoring is worth ~15× more than factual injection.
2. Mandate trajectory outcome annotation. Never allow unannotated execution logs into a skill-distillation pipeline. Label every trajectory with an LLM-as-a-Judge, and target a balanced ~3s2f mix.
3. Deploy two-level retrieval gating at k ≥ 20. Replace flat vector search with domain-bucket routing plus strict programmatic triggers to defeat semantic confusability.
4. Build boundary fallbacks into every skill. Include explicit failure-recovery steps and precondition checks so runbooks degrade gracefully instead of looping — this is the direct countermeasure to the SC3 fault quadrupling.
5. Implement runtime secret scrubbing. Sanitize all stdout/stderr streams and skill files automatically before persistence.
6. Compile MAS to SAS where feasible. Audit high-message-volume multi-agent topologies for compilation into skill libraries to cut token overhead ~54% and latency ~50%.
7. Treat skill use as a lifecycle problem, not a memory-injection mechanism. Audit the catalog like a library of production runbooks: creation quality, retrieval routing, boundary behavior, security hygiene, and retirement.

Enterprise Skill Deployment Checklist

• [ ] Trajectory validation: 100% of historical logs carry success/failure labels before distillation.
• [ ] Secret scrubbing: Automated filters active on stdout/stderr and all persisted skill files.
• [ ] Catalog gating: Two-level gating live for repositories with k ≥ 20 skills.
• [ ] Boundary fallbacks: Explicit failure-recovery steps present in every SKILL.md.
• [ ] Pre-flight checks: Dependency and tool-availability verification embedded at the start of each runbook.
• [ ] MAS compilation audit: High-token multi-agent graphs evaluated for compilation into SAS skill libraries.
• [ ] Governance substrate: Skill execution wrapped in 5R-governed reasoning with thinking-budget caps and telemetry.

12. Conclusion

The 8,135-trial evidence base overturns the industry's default assumptions about agent skills. Skills work not by filling knowledge gaps but by anchoring execution — converting noisy prior experience into deterministic procedural checklists that eliminate the low-level environment failures that quietly kill most agent runs. But this stability is purchased with rigidity: boundary faults rise, and unannotated or insecure distillation pipelines can poison the library faster than it improves.

Reliable production systems therefore treat skills as a managed lifecycle: annotated distillation in, gated retrieval through, governed execution around, and continuous triage and compilation on top. Accumulating more memories is not the answer to agent reliability — building disciplined abstractions, and the architecture to apply them with judgment, is.

References

Primary Sources (verified)

[1] Jiang, Z., Huang, F., Xing, H., Wu, X., Gao, Y., Cao, R., Wang, M., Liu, S., & Li, Y. (2026). Demystifying Agent Skills: Why They Work—Until They Don't. arXiv:2608.14036 [cs.AI], submitted 14 Aug 2026. Princeton University, UC San Diego, Stanford University, University of Southern California, Johns Hopkins University. The core study of this report. Through controlled experiments across multiple benchmarks, agent harnesses, and LLMs, the authors isolate the effects of skill representation, outcome annotation, retrieval difficulty, and cross-framework robustness. They normalize 8,135 trial records and retain 238 valid unique labels from 240 open-coded records, consolidating observations into a taxonomy of three high-level categories and twelve skill-use modes. Headline results: skills improve over workflow memory by 6.06 points in matched comparisons; procedural anchoring accounts for 65.7% of skill cases versus 4.5% for explicit knowledge injection ("skills stabilize action rather than inject missing facts"); as pools grow from 5 to 100, actual-use precision falls from 29.6% to 3.3% while downstream success remains stable — exact ground-truth invocation is neither sufficient nor necessary. Code and artifacts: github.com/zhiyuanjiang04/demystify-agent-skills.

[2] Dong, G., Gao, Y., Li, L., Xu, T., Hua, Y., & Yang, F. (2026). Agent Skills Can Be Harmful: An Empirical Study of Skill-Induced Failures in LLM Agents. arXiv:2608.11888 [cs.AI], submitted 12 Aug 2026. Huazhong University of Science and Technology, Microsoft Research, Microsoft, University of Illinois Urbana-Champaign. Introduces a differential analysis framework that attributes a failure or cost regression to a specific loaded skill by comparing a skill-guided run against a no-skill or semantically matched reference run. Instantiated on SkillsBench and SWE-Skills-Bench, it yields 307 skill-induced failures: 125 functional failures and 182 efficiency regressions. Key findings: (1) Task-Implementation Faults account for 86 of 125 functional failures (68.8%) — seemingly relevant skills cause agents to incorrectly implement or omit required elements — with wrong artifact locations (24 cases) and environment mismatches (13 cases) following; (2) efficiency regressions are not explained by prompt length alone — Excessive Procedure dominates (114 of 182 cases, 62.6%), led by excessive verification (67) and heavy implementation pipelines (30), showing skills often turn validation checklists into cost sinks; (3) the paper ships SkillTriage, a taxonomy-guided attribution tool producing triage reports.

[3] Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., & Anandkumar, A. (2023). Voyager: An Open-Ended Embodied Agent with Large Language Models. arXiv:2305.16291. The foundational precedent for skill libraries: an embodied Minecraft agent that stores skills as executable JavaScript code in a vector database, accumulated through an automatic curriculum with iterative self-verification — establishing skills-as-code as a complementary representation to markdown runbooks.

[4] Credential Leakage in LLM Agent Skills: A Large-Scale Empirical Study (2026). arXiv:2604.03070 [cs.CR]. The first large-scale empirical study of credential leakage in agent skills. From 170,226 skills on SkillsMP (the largest open-source skill marketplace), the authors sample 17,022 via stratified random sampling and apply static analysis (regex and AST-based secret extraction), dynamic sandbox testing with mock credentials, and manual cross-referencing. They identify 520 affected skills harboring 1,708 security issues and derive a taxonomy of 10 leakage patterns — 4 from developer negligence, 6 from deliberate adversarial construction. Notably, 76.3% of cases are cross-modal: they surface only when natural-language descriptions and executable code are analyzed jointly. Companion work: SkillProbe (arXiv:2603.21019) audits skill marketplaces via multi-agent collaboration, documenting the semantic gap between "safe"-sounding documentation and shadow capabilities in code; an ecosystem-scale study of 31,132 skills found 26.1% contained at least one vulnerability across 14 patterns, with script-bearing skills ~2.1× more likely to be vulnerable (see also Agent Skills in the Wild, arXiv:2601.10338, and Malicious Agent Skills in the Wild, arXiv:2602.06547).

[5] Li, X. (2026). When Single-Agent with Skills Replace Multi-Agent Systems and When They Fail. arXiv:2601.04748 [cs.AI], v1 8 Jan 2026, v2 14 Jan 2026. Formalizes MAS-to-SAS compilation as a mapping Φ : M → S where M = ⟨A, G, Π⟩ compiles to S = ⟨a, S_Φ, σ_Φ⟩, distilling each specialized agent's function into skills and trading inter-agent communication for skill selection. Experiments show substantial token and latency reductions at competitive accuracy. Crucially, the paper also identifies the failure mode: skill selection exhibits bounded capacity analogous to human cognition — accuracy degrades as a phase transition, not a smooth decline, driven by semantic confusability among similar skills, and hierarchical routing mitigates the overload. This directly corroborates the retrieval-collapse and two-level-gating findings of [1].

Related Reading

• SkillsBench — Li, X., et al. (2026). SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks. arXiv:2602.12670. The benchmark used in [1] and [2].
• SWE-Skills-Bench — Han, T., et al. (2026). Do Agent Skills Actually Help in Real-World Software Engineering? arXiv:2603.15401. Second benchmark in [2].
• Agent skills survey — A Comprehensive Survey on Agent Skills: Taxonomy, Techniques, and Applications. arXiv:2605.07358.
• From Multi-Agent to Single-Agent: When Is Skill Distillation Beneficial? — Xu, B., et al. (2026). arXiv:2604.01608. Shows skill lift from MAS distillation ranges from +28% to −2% on the same task, motivating principled distillation criteria.
• SKILL.md quality at scale — What Keeps Agent Skills from Being Reusable? Evidence from 138K SKILL.md Files. arXiv:2608.08453. Finds 91.8% of public skills contain at least one defect — mostly ordinary packaging problems (weak routing metadata, bloated bodies) rather than exotic attacks.

A Note on Provenance

The five primary sources above are verified publications. Several framing devices used in this report — FCoT 3.0 with the 5R governance protocol, the six-level Agentic AI Maturity Model with sublevels 6a/6b/6c, and specific synthesis figures such as the −54% token / −50% latency compilation deltas and the 3s2f/78.1% trajectory-ratio table — originate from the synthesis material this report consolidates rather than directly from the cited papers, and should be treated as architectural interpretation layered on top of the verified empirical findings. Where the source material and the published papers disagreed (e.g., the security statistic), this report uses the published figures.