Companion repo: builders-bible/project-05-agent-fleet

Time estimate: 18-25 hours across 6-7 sessions

What this proves you can build: A system where you are the CTO, not the coder — where multiple autonomous agents run their own spec → generate → verify loops, hand work to each other, and ship something you'd defend, with you holding the engineering bar instead of the keyboard.

Why This Project

Every project before this one had you at the keyboard. You wrote the orchestrator, you wrote the prompts, you wired the agents together by hand. The AI was a component inside a system you built.

Part VI inverts that, and this capstone is where you feel the inversion in your hands. Here, the agents don't just answer questions inside your system — they build the system. You direct a fleet that reads a codebase, plans, writes code, reviews its own diffs, runs tests, and reports back. Your scarcest resource stops being typing and becomes specification and verification — exactly the shift Chapter 35 named.

You will build a real, working autonomous multi-agent code reviewer and research-report engine — one repo, two pipelines that share a fleet:

1. The Reviewer pipeline: point it at a pull request or a diff, and a fleet of specialized agents reviews it — one hunts security holes, one checks correctness against tests, one reads deletions for dropped guard clauses, and an integrator synthesizes a verdict with line-anchored comments.
2. The Research pipeline: give it a question, and a fleet fans out across sources, fetches and reads them, adversarially verifies each claim against a second source, and synthesizes a cited report — refusing to assert anything it couldn't ground.

Both pipelines run on the same orchestration spine. By the end, you'll have lived every chapter of Part VI: you'll author an MCP server the agents call (Ch 36), let an agent drive a browser to fetch a source (Ch 37), engineer the context each agent carries across steps (Ch 38), gate every output behind an eval (Ch 39), and dispatch the whole fleet in parallel with a verify-then-commit barrier (Ch 40).

This is the moment you become the contractor, not the carpenter.

Prerequisites

Before Session 1, you should have:

• Finished Project 04, or be comfortable with model routing, cost tracking, and a hand-rolled orchestrator. This project assumes that foundation and builds up from it.
• Read Part VI, Chapters 35-40. Each phase below maps to a chapter; the project is the lab for the lectures.
• A working coding-agent setup (Claude Code or equivalent) with an API key and a spend cap you've actually set.
• An MCP-capable runtime — you'll author one server in Phase 2 (Ch 36) and your agents will call it.
• A real repo to point the reviewer at. Use one of your earlier projects. Reviewing your own code is the only way you'll catch when the fleet is wrong.
• A $2 budget cap for the whole project, enforced in code, not in your head. A runaway fleet is the fastest way to a surprise bill.

One honest gate before you start: if you cannot yet read a diff and say why a deletion is dangerous, do that drill first (Ch 35's diff section). A fleet that builds faster than you can verify is not leverage — it's a liability with good commit messages.

Architecture Overview

+--------------------------------------------------------------+
|                       THE CONDUCTOR                          |
|     (you specify; it dispatches, gates, and commits)        |
|                                                              |
|  Input: a TASK SPEC (review this diff | research this Q)     |
|  Holds: the shared CONTEXT BUDGET + the COST CAP             |
|                            |                                 |
|         +------------------+------------------+              |
|         v                  v                  v              |
|  +------------+    +------------+    +------------+          |
|  | WORKER A   |    | WORKER B   |    | WORKER C   |          |
|  | (its own   |    | (its own   |    | (its own   |          |
|  |  spec->    |    |  spec->    |    |  spec->    |          |
|  |  gen->     |    |  gen->     |    |  gen->     |          |
|  |  verify    |    |  verify    |    |  verify    |          |
|  |  loop)     |    |  loop)     |    |  loop)     |          |
|  +-----+------+    +-----+------+    +-----+------+          |
|        |                 |                 |                 |
|        +--------- VERIFY BARRIER ----------+                |
|        (no worker output is trusted until                   |
|         it passes the eval gate; failures                   |
|         route back to the SPEC, not retry)                  |
|                          |                                  |
|                          v                                  |
|                +------------------+                         |
|                |    INTEGRATOR    |                         |
|                | (synthesize the  |                         |
|                |  verified parts  |                         |
|                |  into one        |                         |
|                |  deliverable)    |                         |
|                +------------------+                         |
|                                                             |
|  Cross-cutting: MCP TOOLS | EVAL GATE | COST METER | LOG    |
+--------------------------------------------------------------+

The shape to notice: every worker runs the same loop you learned in Chapter 35 — spec, generate, verify — but now there are several running at once, and a barrier sits between "generated" and "trusted." Nothing reaches the integrator on faith. That barrier is the whole reason a fleet ships reliable work instead of a fast-rising tide of plausible garbage.

Tech stack:

• Orchestration: a hand-rolled conductor (a dispatcher, a state object, a barrier) — not a framework. When the fleet deadlocks at 2 AM, you debug what you can see (Ch 40).
• Workers: coding agents run as subprocesses, each given a scoped spec and a fresh context (Ch 38).
• Tools: one MCP server you author (Ch 36) exposing fetch_url, read_file, run_tests, post_comment.
• Browser worker: one agent drives a real browser for sources behind JS, sandboxed and human-gated (Ch 37).
• Evals: LLM-as-judge + rule checks as the gate (Ch 39).
• Models: route by stakes — cheap model for fan-out and classification, strong model for the security review and the final synthesis.
• State + cost: SQLite for run state, a token meter wrapping every call. No surprise bills.

What You'll Build, Step by Step

Phase 1: The Conductor and the Verify Barrier (Session 1)

Before a single worker exists, you build the spine. The conductor is not magic — it's a dispatcher, a shared state object, and one non-negotiable rule: no worker output is trusted until it passes the gate.

You'll implement:

• Dispatch: take a task spec, split it into scoped sub-specs, hand each to a worker.
• The barrier: collect worker outputs, but mark every one UNVERIFIED until the eval gate (Phase 5) stamps it.
• Failure routing: when a worker's output fails the gate, the conductor does not re-run the same prompt. It routes the failure back to the sub-spec — the Chapter 35 reflex, now enforced in code. A retry against an unchanged spec is a slot-machine pull, and your conductor refuses to pull.
• The cost meter: every call increments a shared counter; at the $2 cap, the conductor halts the whole fleet, no exceptions.

Build this as plain code you fully understand. The fleet is only as trustworthy as the spine, and you cannot be accountable for a spine you can't read.

Checkpoint 1: Feed the conductor a fake task that fans out to three no-op workers. Confirm: outputs arrive marked UNVERIFIED, a forced gate-failure routes back to the sub-spec (you see the spec change in the log, not a blind re-run), and tripping the cost cap halts all workers, not just one. If the cap doesn't halt the fleet, stop and fix it now — everything downstream spends money.

Phase 2: The MCP Tool Server (Session 2)

Your workers are blind until they can reach the world. In Chapter 36 you learned to author an MCP server; here you ship one. It exposes exactly four tools, each tightly scoped:

• fetch_url(url) — returns cleaned text, with a domain allowlist and a byte cap (no agent pulls a 2 GB page).
• read_file(path) — read-only, jailed to the target repo, so a worker can't wander into ~/.ssh.
• run_tests(cmd) — runs the project's test command in a sandbox, returns pass/fail + output, with a timeout.
• post_comment(line, text) — the reviewer's only write tool; it stages a comment, it does not push to anyone's PR.

The discipline here is least privilege. Every tool is a hole in your sandbox. read_file that isn't jailed is a data-exfil path; run_tests without a timeout is a hang; fetch_url without an allowlist is an SSRF waiting to happen. You're not just enabling the fleet — you're deciding what it's forbidden to do.

Checkpoint 2: From a throwaway agent, call each tool. Confirm the negatives: read_file("/etc/passwd") is refused, fetch_url to an off-allowlist domain is refused, run_tests on an infinite loop times out and returns rather than hangs. The tools that don't work are as load-bearing as the ones that do.

Phase 3: The Reviewer Fleet (Session 3)

Now the first real pipeline. Point the conductor at a diff, and it dispatches three specialist workers in parallel, each with a distinct sub-spec so they don't all find the same thing:

• The Security worker (strong model, high-verify): hunts injection, auth bypass, secrets, SSRF. Stakes are systemic, so this one gets the expensive model and the strictest gate.
• The Correctness worker (mid model): reads the diff against the tests, runs run_tests, flags logic that the happy path hides.
• The Deletions worker (mid model): does one thing — reads every minus line and asks "what guard, null check, or error case quietly vanished?" This is Maya's 2 AM bug from Chapter 35, turned into a dedicated agent.

Each worker emits findings via post_comment, anchored to specific lines, marked UNVERIFIED. The point of three narrow specs instead of one "review this code" is the same point as three narrow employees instead of one generalist: scoped specs collapse whole branches of vague output.

Checkpoint 3: Plant three known bugs in a test diff — a hardcoded secret, an off-by-one the tests catch, and a deleted null check. Run the fleet. All three workers should surface their respective planted bug with the right line number. If the Deletions worker misses the dropped null check, your sub-spec was too vague — fix the spec, not the model.

Phase 4: The Research Fleet (Session 4)

Same spine, different workers. Give the conductor a question and it fans out:

• Fan-out workers (cheap model): each takes one sub-query, calls fetch_url, and extracts candidate claims with their source URLs.
• One Browser worker (Ch 37): for a source behind JavaScript, this agent drives a real, sandboxed browser — human-gated, so you approve before it acts on any page that isn't read-only. Computer-use is powerful and unforgiving; the human-in-the-loop is not optional.
• The Verifier worker (strong model): the adversary. For every claim, it must find a second, independent source that confirms it — or the claim is dropped. No single-source assertions survive.

This is retrieval-as-memory and adversarial verification working together. The fleet's job is not to sound confident; it's to refuse to assert what it can't ground twice.

Checkpoint 4: Ask a question with one well-known true answer and one common misconception. Confirm the report states the true answer with two sources, and omits or flags the misconception because the Verifier couldn't find a second credible source. A research fleet that repeats a popular myth confidently has failed, no matter how fluent the prose.

Phase 5: The Eval Gate (Session 5)

This is the barrier from Phase 1, now given teeth. Nothing reaches the integrator until it clears the gate. The gate is a mix, per Chapter 39:

• Rule checks (cheap, deterministic): every research claim has >= 2 source URLs; every review comment has a valid line anchor; no comment references a line outside the diff (the hallucinated-line check — the agent's favorite slop).
• LLM-as-judge (for what rules can't see): is this finding relevant and actionable, or is it the fleet gold-plating — flagging style nits on a security review?

The gate can pass, reject (route back to the sub-spec), or escalate to you. Wire the escalation: when the gate is unsure, it stops and asks the human, because the worst outcome isn't a slow gate — it's a confident one that's wrong.

Checkpoint 5: Run a known-good task end-to-end and confirm it passes the gate cleanly. Then hand-corrupt one worker's output — a comment on a line that isn't in the diff — and confirm the gate rejects it and routes back, rather than letting it through to the integrator. Watch the eval fail before it passes. That's the strongest rung on the verification ladder, and it's the one you build your trust on.

Phase 6: The Integrator and Verify-Then-Commit (Session 6)

The integrator takes only the verified parts and synthesizes one deliverable — for the reviewer, a single ranked review with CRITICAL findings first and informational ones second; for research, one cited report.

Then the barrier that makes a fleet safe: verify-then-commit (Ch 40). The integrator's output is itself gated one final time, and only then is the side effect allowed — the review comment posted, the report written to disk. Nothing the fleet does becomes real until it has been proven, and even then the irreversible step (posting to a live PR) waits for your one-key approval. Reversible side effects, the fleet commits. Irreversible ones, you do.

Checkpoint 6: Run both pipelines end-to-end on real inputs. Confirm the integrator surfaces CRITICAL before INFORMATIONAL, that the final output is gated again before any side effect, and that the one irreversible action (a live PR comment) pauses for your approval. Trust the process; verify the output; own the irreversible step.

Phase 7: The Conductor's Dashboard (Session 7)

You cannot direct a fleet you cannot see. Build a small dashboard that shows, per run:

• The fleet timeline: which worker ran when, in parallel, and how long each took.
• The verification trail: for every output, did it pass, reject-and-respec, or escalate — and how many loops it took.
• The cost breakdown: tokens and dollars per worker, so you see (as in Project 04) that fan-out's many small calls can outspend one big synthesis.
• The respec log: every time a failure routed back to a spec, what changed. This log is your curriculum — it shows you the exact shapes of ambiguity that bite you, the way Chapter 35's exercise promised.

Checkpoint 7: After ten runs, open the respec log and find the cluster. Two or three kinds of ambiguity will account for most failures. Fix those in how your sub-specs are written. Your hit rate will jump more than any model upgrade could deliver — because you just upgraded the specifier, which is you.

What Will Go Wrong (And What It Teaches)

• Two workers find the "same" bug and the integrator reports it twice. Your sub-specs overlapped. This teaches you that parallel agents need non-overlapping mandates — the org-chart discipline of Chapter 35, that a team works only when roles are distinct.
• A worker passes the gate with a confident, plausible, wrong finding. Your LLM-as-judge agreed with a fluent lie. This teaches you why rule checks sit alongside the judge — a deterministic check doesn't care how confident the prose is.
• The fleet deadlocks: every worker waits on the barrier, nothing finishes. A worker crashed and never reported, so the barrier never releases. This teaches you that orchestration code needs timeouts and a dead-worker reaper more than the workers need cleverness.
• Cost triples on a "simple" run. A worker hit the gate, failed, re-spec'd, and looped — three times. This teaches you circuit breakers: a max-respec count per sub-task, after which the conductor escalates to you instead of spending again.
• The browser worker clicks something it shouldn't. It found a "Delete" button and, being helpful, used it. This teaches you — viscerally — why computer-use is human-gated on any irreversible action (Ch 37). Read it once in a book; feel it once in a sandbox; never forget it.

Stretch Goals

For the reader who wants to push past the bar:

• A reviewer-of-the-reviewer. Spawn a second, blind agent (no fleet context, only the diff and the fleet's verdict) to find flaws in the fleet's own review. This is the "stress test" move — adversarial verification turned on yourself.
• Dynamic dispatch. Let the conductor decide how many workers to spawn based on diff size or question complexity, instead of a fixed three. Small diff, one reviewer; sprawling refactor, five. (Ch 40's dynamic dispatch, made real.)
• Persistent fleet memory. Give the fleet a memory store (Ch 38) so a reviewer remembers the conventions it learned on this repo last week — and stops re-flagging the same intentional pattern.
• Self-authored sub-specs. The hardest one: let a planning agent write the sub-specs for the workers from your one-line task, then you approve them in plan-mode before dispatch. You become the editor of specs, not the author — one level up again.
• A second tool surface. Add a git_blame MCP tool so the reviewer can cite who last touched a risky line and when, turning a finding into a conversation.

Definition of Done

Point the reviewer fleet at a real pull request — ideally one on your own earlier project, deep enough that you'd catch a subtle miss. Then run the research fleet on a question you know cold. For both:

1. Every finding or claim that reaches you is verified — review comments anchor to real lines; research claims carry two independent sources. Zero hallucinated line numbers, zero single-source assertions.
2. The fleet caught at least one thing you would have missed on a tired read — a dropped guard, an unsourced "fact."
3. The total cost stayed under your $2 cap, and the dashboard shows you exactly where every cent went.
4. The respec log proves the fleet routed at least one failure back to a spec, not to a blind retry.
5. The one irreversible action waited for your approval — the fleet never committed something it couldn't undo without you.

Then the real bar, the one that means you've crossed into Part VI for good: hand the reviewer's verdict to an engineer you respect and have them check the fleet's work. They should find that the review is one they'd have been glad to receive — not slop they have to clean up, but genuine findings, ranked, defensible, with the CRITICAL ones first.

That's the bar. Not "the agents ran." A fleet ran, you directed it, and you can stand behind what it shipped — because you specified what correct meant, and you proved it got there. The keyboard was never the point. Direction was.