The Wrong Mental Model of Success

Yesterday I sat down to work on sovereign-agent v0.1.0, an agent framework with tickets, manifests, atomic state transitions, and 148 passing tests. Three example scenarios, all green in CI. Then I ran make example-reviewer-real against a real LLM for the first time.

The agent produced a perfectly-formatted code review of code that does not exist. Functions named add, multiply, divide — none of them in the source file. The framework reported success. Manifest verified. Complete. Every structural guarantee held.

The output was pure fiction.

The question this lesson opens with

Ask yourself, honestly: if your agent framework reports ✓ on every check it knows how to run, what has actually been verified?

Most tutorials never make you answer that, because most tutorials never connect an agent to a real LLM long enough to watch it fail in a way that looks like success. That's what this five-lesson arc is built around — not "how to build an agent framework," that's the backdrop, but what you only learn by watching real failures happen and fixing them in front of a working system. Every failure mode in this arc happened to me in the 18 hours before I taught it. You'll see the traces. You'll see the fixes.

Here's the shape of the five lessons ahead, so you know what you're walking into: the architecture first (this lesson and the next two — the two-halves split, then the eight decisions behind it), then the journey (the eight real failures that architecture didn't prevent on its own), then the live proof (a real demo against real models, plus the dataflow-integrity pattern that closes the gap this lesson opens on), and finally the production honesty pass (what v0.1.0 still gets wrong, on purpose, and what to do this week). Each lesson stands on the vocabulary the one before it built — that's why the prerequisite chain matters here more than in most courses.

How most tutorials draw an agent

You give an LLM some tools. It calls them in a loop until it's done. That's the picture almost every course opens with, and it isn't wrong — it's just so incomplete that it hides every question that actually decides whether the thing survives production.

Look at the picture and try to answer any of these:

• When the LLM runs the same workflow twice, where does prior knowledge live?
• When it crashes halfway through, how does the next run know what already completed?
• When two workflows run in parallel, how do they avoid stepping on each other's outputs?
• When the LLM hallucinates a tool call, what happens?
• When a tool call succeeds but with the wrong argument, who notices?
• When the output file exists but contains lies, who notices?

The loop-with-tools picture has no answer for any of these. It's drawn at the wrong level of abstraction — it shows you the part that's easy to build and is silent about the part that decides whether you can trust what comes out.

What sovereign-agent actually built instead

sovereign-agent organizes everything around sessions as directories. Every run creates sessions/sess_<12hex>/, and everything that run touches lives inside that one path:

This isn't a novel idea original to a teaching framework built in 18 hours. It's the convergent answer that Claude Code, OpenHands, Aider, SWE-agent, and Devin all arrive at independently: scope everything — memory, IPC, tool state, workspace, logs, the system prompt — to one directory per session. Crashes recover by ls. Concurrency is bounded by locking at the session level. Debugging is cat. Audit is tar. If you take one structural idea from this arc, take that one; the rest of this course builds on it.

Here's what that buys you concretely, compared to the tutorial-level answer:

Debugging by cat, and where it stops helping

The night before this class, I debugged a failed session from exactly one file:

$ cat sessions/sess_5ab10359c72a/logs/trace.jsonl
{"event_type": "executor.tool_called", "payload":
 {"tool": "read_file", "arguments": {"path": "workspace/source.py"},
  "success": false, "summary": "file not found"}}
{"event_type": "executor.tool_called", "payload":
 {"tool": "list_files", "arguments": {"path": "workspace"},
  "success": true, "summary": "0 entries"}}
{"event_type": "executor.tool_called", "payload":
 {"tool": "complete_task", "arguments":
  {"result": {"status": "failed",
              "reason": "No source file found in workspace"}}}}

Three lines and the failure is obvious: the LLM never called the analyzer tool at all. It went looking for files that weren't there, gave up, and marked the session failed. That's cat doing real diagnostic work — the session directory earns its keep here.

But notice what that trace can't tell you. It can tell you a session failed cleanly. It cannot tell you whether a session that reported success actually did the right thing. That's the trap the make example-reviewer-real run walked straight into: every structural signal — ticket written, manifest hashed, state advanced to complete — held, on a review of code that was never there. The framework can guarantee that operations happened, that files weren't silently corrupted, that state moved forward and not sideways. It cannot guarantee that what the LLM said about its own output was true. That gap — structural correctness versus semantic correctness — is what the rest of this arc is about closing, one failure at a time.

What to carry into the next lesson

The gap you just watched — a framework telling the truth about its own operations while the LLM's output is fiction — doesn't get closed by a smarter model or a longer prompt. It gets closed by architecture: knowing which decisions a language model should never be trusted to enforce, and building a half of the system that doesn't ask it to.