Shippy's Agent Lessons: Why Memory Goes Stale

Anyone running a multi-agent system in their own production knows this scene: the leader agent hands a task to a worker, and the worker either builds the right command or, with total confidence, fabricates a wrong — but plausible-sounding — call. The difference usually isn’t in the model’s intelligence; it’s in how predictable the tools you put in the agent’s hands are. So, like everyone hunting for agent-building lessons, I read Ai2’s write-up on Shippy, its maritime-safety agent, against AgentSpace’s leader+worker Claude agents.

What Shippy describes

Ai2’s Skylight team described Shippy’s architecture on the Hugging Face blog. The agent is built from three parts: a system prompt that draws the behavioral boundaries (“soul”), task instructions in markdown+frontmatter format (“skills”), and a separate “config” layer that carries the model/harness choice. Instead of calling the API raw, the agent is wrapped in a custom CLI that exposes typed commands. The team sums it up: “Agents are nondeterministic. You can’t control what the model decides to do, but you can make the tools it reaches for predictable.”

Our gap: no formal eval, just eyeballing

The part that caught me most wasn’t the CLI, it was the eval system: domain experts write a weighted rubric for each task, an LLM judge scores every criterion and notes its reasoning, and the weighted total turns into a pass/fail decision against a fixed threshold. A version that regresses never reaches the user.

There’s no such harness in AgentSpace. The leader agent and I review sprint output by hand; there’s no formal rubric or fixed threshold, and we usually notice memory has gone stale only when something breaks. I’m thinking about building this, but I haven’t yet — right now all I have is intent, not measurement.

Tool determinism and model selection

The second lesson is about CLI determinism: Shippy writes its results to disk rather than the shell, heading off pipe limits and corrupted output up front. In AgentSpace’s headless workers, tool calls aren’t bounded that tightly; I want the agent to read which commands actually exist from a fixed list rather than its own memory, but I haven’t disciplined this as much as Shippy’s CLI layer.

The “route simple queries to a smaller model” idea on their roadmap is adaptable to us too: we already allocate per-sprint budget by the work rather than the person, but I haven’t looked systematically at which worker step could get by on the cheap model — that’s the next experiment.

The takeaway for the reader: rather than making the agent smarter, fixing every tool it touches and every scoring criterion in advance determines reliability more than the model itself does.

What you can do today:

• List the points where a worker agent can freely fabricate a command/API call
• Reduce each point to a single, typed CLI/function call — don’t let the agent invent the encoding
• At the end of a sprint, hand-score at least a few tasks against a weighted rubric (accuracy, out-of-scope suggestions, source violations)
• Note every moment the agent invents “a command that doesn’t exist” — your next skill/tool boundary belongs there