Category: Multi-Agent Systems in Production

Classical software engineering is a century-long campaign to eliminate unpredictability. We write f(x) = y, test it, and expect the same output every time. Agentic software inverts that premise. At its core sits a large language model that does not compute answers so much as sample them from a probability distribution over tokens — P(w_t | context). Any single run is one draw from that distribution. Emergent Engineering is the discipline of building deterministic frameworks — context, harnesses, policies, and observability — around that stochastic core so that reliable, governable behavior emerges from it.

The most useful mental model is a maturity ladder. Each level adds a new capability, and — this is the part teams miss — each new capability introduces a new class of failure that the previous level’s tooling cannot contain. You mature by adding the governance mechanism that tames the new failure. This post walks the six levels of Agentic Governance Maturity, and for each one it does two things: critiques the level’s shortcomings, and lays out exactly how to build the next level. The interactive portal below lets you explore each level, run a policy decision, and watch a swarm’s entropy live.

The shift: from control to containment

Three eras, side by side. In classical software the author holds absolute control: explicit logic, deterministic output, f(x) = y. In machine learning we surrender the exact function and approximate it from data — f̂(x) ≈ y — trading authorial control for pattern-matching at scale. In emergent AI even the approximation is probabilistic: the model emits a distribution over next tokens and we sample it.

The engineering consequence is the whole game: you cannot unit-test your way to a guarantee when the unit is stochastic. So you stop trying to make the core deterministic and instead govern it. You engineer the boundary around the model — what it can see, what it can do, and what it is forbidden from doing — with the rigor classical software reserved for the logic itself. Determinism doesn’t disappear; it relocates to the boundary. The six levels below are six successively stronger boundaries.

The Agentic Governance Maturity model

Progress in agentic systems is less about larger models than about the sophistication of the governance we build around them. Six levels trace that path. Each introduces a distinct engineering challenge, a control mechanism, a set of shortcomings, and a concrete path to the next level.

Level 1 — Prompting: carving a persona out of latent space

A pretrained model is a superposition of every persona, register, and reasoning style in its training data. Prompting is the act of collapsing that superposition toward one region with declarative language: a role (“you are an SRE telemetry sentinel”), an output contract (a JSON schema), a handful of exemplars, and explicit negative constraints (“never mutate production”). The governance mechanism here is the instruction contract itself.

Shortcomings. Everything lives in one editable text field, and that is the whole problem. Instructions drift as the conversation grows and the original system prompt slides out of the model’s effective attention. A later user turn can override an earlier rule — classic prompt injection — because to the model, instructions and data are the same tokens. With no grounding, the model fills gaps by inventing plausible facts. And structurally it is blind and handless: it cannot reach fresh or private data, cannot act, and remembers nothing beyond the window. At Level 1 your rules are suggestions, not walls.

Reaching Level 2. Stop feeding the model facts by hand and build a retrieval layer that does it automatically. Concretely: index your private and live data (chunk it semantically, embed it, store the vectors); insert a retrieve → assemble → generate step so every request first pulls the most relevant context and injects it into the prompt with citations; and add a memory tier so state survives across turns. The moment the model reasons over data it fetched rather than data you pasted, you are at Level 2.

Level 2 — Context: grounding a frozen model in a live world

A model’s weights are frozen at training time and generic to the whole internet. Level 2 injects the specific, current state it needs at inference time — the essence of Retrieval-Augmented Generation and, more broadly, dynamic state hydration. Because the context window is finite and every token costs latency and money, this level is really an information-retrieval problem wearing an LLM hat: chunking strategy, hybrid dense + lexical (BM25) retrieval, a cross-encoder reranker, and explicit context assembly.

Shortcomings. Retrieval is now a dependency, and it fails quietly. Low recall silently omits the one document that mattered, and you never see the omission. Context dilution — the “lost in the middle” effect — buries the key fact among filler so the model ignores it. A stale index confidently grounds the model in yesterday’s truth. A poisoned document becomes an injection vector that survives into the prompt. And the ceiling remains hard: context makes the model an authoritative observer, but it is still read-only. It can tell you exactly which pod to restart; it cannot restart it.

Reaching Level 3. Give the model hands — but typed ones. Define a small set of tools as strict schemas (JSON Schema / Pydantic, or expose them over MCP), then build a harness that stands between the model and every side effect: it parses the proposed call, validates the arguments, authorizes the identity, executes in a sandbox, and returns a structured result. Start with read-only tools to prove the loop, then add mutating tools one at a time behind role-based access control. When the model can act and the harness is what makes that safe, you are at Level 3.

Level 3 — Harness Engineering: where stochastic meets deterministic

This is the most important level and the one teams most often skip. A model that can act emits a string claiming to be a tool call; the harness is the typed, sandboxed layer that parses it, refuses anything that fails validation, executes the rest inside strict boundaries, and feeds a structured result back. Protocols like MCP standardize the wire format, but the harness is where you re-impose determinism on a probabilistic caller — three responsibilities, always in this order:

class AegisHarness:
    def __init__(self):
        self.allowed = ["staging", "read-only"]
    def execute_tool(self, call_json):
        # 1. SCHEMA VALIDATION — is this a well-formed, typed call?
        call = ToolCall.model_validate_json(call_json)   # reject malformed args
        # 2. RBAC ENFORCEMENT — is this identity permitted this namespace?
        if call.namespace not in self.allowed:
            return {"status": "blocked", "reason": "RBAC: namespace not permitted"}
        # 3. SANDBOXED EXECUTION — run with least privilege, timeouts, idempotency
        return sandbox.run(call, timeout_s=30, idempotency_key=call.id)

The discipline in one line: the model proposes; the harness disposes. Its toolkit is strict argument schemas, tool-layer RBAC, least-privilege sandboxing, idempotency keys, and structured errors the model can recover from.

Shortcomings. The harness makes a single action safe, but it has no memory of the last action or intent for the next. It cannot express “do A, then B only if A succeeded, and roll back both if C fails.” Push a multi-step goal through a bare harness and you get brittleness: the model re-plans from scratch on every turn, loses track of what it already did, repeats or contradicts earlier steps, and leaves the system half-changed when one call fails. A validated action is necessary but it is not a workflow.

Reaching Level 4. Lift control flow out of application code and into an explicit orchestrator. Model the task as a state graph (a DAG) with a typed state object passed between nodes; let the model produce the plan while a graph engine (LangGraph-style) executes it. Add checkpointing so a long run can resume, parallel fan-out/fan-in for independent branches, and saga-style compensation so a mid-run failure can undo prior steps. When a durable, recoverable graph — not your if/else — drives the sequence, you are at Level 4.

Level 4 — Loop Engineering : Sequential: orchestrating stateful, branching work

Level 4 chains individual tool calls into a multi-step workflow. The model produces a plan and an orchestrator executes it as a stateful graph, holding intermediate results in a “state fabric,” running independent branches in parallel, and joining them back together. The unit of reasoning is no longer a single call but a transaction that spans several — and that is exactly what changes the failure surface.

Shortcomings. You have inherited every hard problem of distributed systems, now driven by a stochastic planner. Partial failure is the signature bug: step 3 of 5 throws and the system is left half-mutated. Parallel branches race on shared state. Context is lost across a hand-off, so a later node reasons on stale intermediate results. And more subtly, the plan itself is fixed at authoring time: a hand-drawn graph can only handle the paths you anticipated, so the instant reality diverges from the diagram, the workflow is stuck. Better prompts do not fix any of this; idempotency, transactions, and compensation do.

Reaching Level 5. Hand the planning to the agent and wrap it in governance. Replace the fixed graph with a closed perceive → decide → act → observe loop (ReAct or plan-execute) that chooses its own next action and reacts to results — then bound it: explicit termination conditions, hard iteration and token budgets, and reflection between steps. Critically, stand up an out-of-band policy engine that evaluates every proposed action before it runs, plus human-in-the-loop gates on irreversible operations. When the agent decides the plan and a policy engine — not the prompt — governs it, you are at Level 5.

Level 5 — Autonomy Engineering with the Autonomous Loop: policy engineering for self-directed agents

At Level 5 the agent enters a continuous cycle, choosing its own next action until a goal or termination condition is met. The per-step validation from the harness is necessary but no longer sufficient; you need standing rules about what ought to happen. That is policy engineering, and its logic is deontic: Permissions, Prohibitions, and Obligations, evaluated out of band by a policy engine before any action runs.

# Out-of-band deontic policy — evaluated by the engine, NOT the model.
# PROHIBITION (F): no production mutation during the Friday change-freeze.
deny[action] {
    action.verb    == "patch"
    action.target  == "production"
    time.weekday   == "Friday"
}
# OBLIGATION (O): any allowed prod change must emit a health-check after.
require_healthcheck[action] { action.target == "production" }

The essential move is to keep policy outside the model. A rule in the system prompt is a suggestion the model can be argued out of or injected past; a rule in a separate engine is a wall. The agent may correctly reason that patching the database fixes the incident — and still be blocked because it’s Friday. The reasoning is real; the policy is what makes it governable. (Try the simulator in the portal above: set the namespace to production, check “Is Friday,” and watch the prohibition roll it back.)

Shortcomings. A single governed agent is still a single agent — a bottleneck and a single point of failure. It has one perspective, so it is prone to confident, unchallenged mistakes; nothing in the loop plays devil’s advocate. Its policy engine keeps it safe but does not make it smarter, and complex problems that need parallel specialists — one to detect, one to fix, one to attack the fix — do not fit inside one linear loop. Scale it by simply running more copies and they collide, because there is no protocol for them to coordinate.

Reaching Level 6. Decompose the one agent into specialized roles and replace central orchestration with decentralized coordination. Give them a shared blackboard where they publish observations and bid on tasks (a contract-net market), and add explicitly adversarial roles — a critic, an adversary — so the group challenges its own conclusions instead of amplifying them. Then change what you observe: with no central trace, instrument aggregate signals (entropy, consensus stability) and wire an autonomy governor that throttles the swarm when those signals go out of bounds. When useful behavior comes from the interaction rather than any one agent, you are at Level 6.

Level 6 — Collaborative Swarm: emergence engineering

At the top level there is no central orchestrator to inspect. Multiple specialized agents — say a Sentinel that detects symptoms, a Mutator that proposes fixes, and an Adversary that stress-tests them — interact through a shared medium, and the useful behavior is authored in none of them. It is a property of the interaction. You stop programming steps and start engineering the conditions and boundaries under which good behavior emerges. Because there is no trace to read, you measure the system’s physics: agentic entropy (HA), how disordered the collective’s decisions are, and consensus stability, how reliably it converges.

Shortcomings. Emergence cuts both ways — the same interactions that solve problems no single agent could also produce failures no single agent intended. Swarms oscillate and fail to converge; they slide into echo-chamber groupthink when the adversarial roles are too weak; a fault in one agent can cascade through the shared blackboard; and agents can settle into emergent collusion that no one designed and no trace explains. Debugging is genuinely hard, because there is no linear story to replay — only aggregate signals to interpret.

Beyond Level 6. There is no Level 7 — the frontier is tighter governance of emergence itself. The governing move is the “systemic chill-out” policy: when entropy crosses a safe threshold, the observability harness automatically scales down the swarm’s autonomy until order returns. From here the work is better entropy and consensus metrics, formal convergence guarantees, and federation across swarms. We do not merely watch emergence; we engineer its limits. That is the enduring role of the human operator in the age of emergent AI — not to author every action, but to set the physical boundaries within which useful behavior is allowed to emerge.

Practitioner takeaways

• Locate yourself on the ladder before you optimize. Most teams over-invest in prompt cleverness (Level 1) and under-invest in the harness (Level 3) that actually makes agents safe to ship. Effort should follow the failure class you’re actually hitting.
• Don’t climb a level until you’ve contained the one below. Each level’s failure class is only tolerable because the previous level’s governance mechanism is solid. An autonomous loop on top of an unvalidated harness is an outage generator.
• Determinism belongs at the boundary, not the core. Stop fighting the model’s stochasticity; spend that budget on retrieval quality, schema validation, RBAC, sandboxing, and policy.
• Keep policy out of the model. Prohibitions and obligations enforced by an out-of-band engine survive prompt injection and model drift. Rules “in the system prompt” do not.
• Instrument for emergence before you need it. Entropy and consensus metrics are cheap to add early and impossible to reconstruct after an incident.

Agentic governance is not a rejection of software rigor — it is that rigor, relocated. We moved it from the logic to the boundary, one level at a time. The core became probabilistic; our discipline did not.