AI Agent Tool Permissions Sandbox: An Agent Testing Its Own Output
How does an agent test its own output?
A while back I got stuck on a small detail in the datasette-apps project: a function called app_debug() that lets an agent run and test the app it just generated, inside a limited sandbox. It looks like a minor implementation detail, but it actually touches the core problem of AI agent tool permissions sandbox design: once you tell an agent “go try this,” what does it actually have access to during that trial run? The whole system, or just the thing it built?
That question felt familiar because I hit the exact same wall building AgentSpace. Worker agents pick up real tasks, generate code, run things — and the most reliable way to verify the output is to have the agent itself say “run this now, tell me what happened.” The problem: which permissions does it need for that, and which ones should it absolutely not have?
What Datasette solved
The logic in datasette-apps roughly works like this: an agent produces an app or a piece of code, then tries it out not in production and not in its own main session, but in a separate, bounded execution context. app_debug() acts like a preview gate — the agent gets to see whether what it built actually runs, but that act of seeing doesn’t grant write access to the rest of the system.
The elegant part is that they built the sandbox as a separate permission set, not necessarily a separate machine. It’s not really about disposable VMs or containers (I compared that angle separately when I looked at clawk’s approach) — it’s about which tool calls are open to the agent at test time. That distinction matters, because isolation and permission are actually two different problems, and conflating them makes the design needlessly complicated.
Our problem: letting a worker test its own output
In AgentSpace, a lead agent hands off a task to a worker, the worker produces code during a sprint and updates the task board. The real question here: when the worker tests its own code to see “does this work,” what permissions is it operating under?
Why blast radius is the actual issue
Telling a worker agent “test your own output” looks like it hands over one bundle of permissions, but it’s actually two:
- Permission to execute the code (necessary — that’s the whole point of the test)
- Permission to touch external systems during that execution (usually unnecessary, but left open by default)
The second one is where things go wrong. When a worker says “I’m testing this API integration,” does it actually need to hit an external service, or can it validate against a mock/stub response? Most of the time the latter is enough, but agents (and we ourselves) default to the “test against the real thing, it’s the safest bet” reflex. That reflex unnecessarily widens the blast radius — the area a mistake can spread into.
A sandbox is not the same thing as a permission list
The real lesson I took from the datasette-apps example: a list of “these tools are allowed, these aren’t” isn’t enough on its own. Because the same tool — writing a file, making an HTTP request — carries a different risk depending on whether it’s used in a test context or a live task context. A sandbox adds a “context” layer on top of the tool permission list: the same worker can use the same tool at two different scopes depending on which context it’s in.
I described the tool/permission layer in AgentSpace before, when I went through Claude Skills — the logic there was binding permissions to a tool. The extra layer here is binding permissions to a context (test vs. live task) — and I haven’t systematically separated that yet. Right now the line between a worker’s “test mode” and “production mode” isn’t explicit in code; it’s mostly enforced through rules baked into the task description.
Where AgentSpace stands today
To be direct about it: we don’t have a separate sandbox function like datasette-apps’ app_debug() yet. Worker agents pick up tasks from the task board, write to their persistent memory, run on Claude, and mostly test their own output inside the same session, with the same tool permissions. This works, because sprints are relatively small in scope — but “it works” and “it’s designed correctly” aren’t the same thing, and I know that.
The concrete idea I’m taking from Datasette’s approach: mark test-run execution with a different tool permission profile than live-task execution. So the moment a worker says “I’m now verifying my own output,” the tool set it gets should automatically narrow — write permissions collapse to a read-only copy, external calls drop to mocks. I haven’t built this yet; it’s the plan for now.
What I still don’t know
- Whether this split (test mode / task mode) should be enforced at the task-definition level or at the tool layer — I haven’t decided.
- If I make the sandbox too strict, there’s a risk the worker misses real integration bugs that a mock can’t catch — I haven’t measured that trade-off.
- I haven’t read datasette-apps’ actual implementation line by line, only borrowed the concept — whether I end up building the exact same architecture is still open.
This post isn’t a solution announcement, it’s a problem statement. The real question behind searching for “AI agent tool permissions sandbox” is usually this: how do you build trust in an agent without handing over full authority. Datasette’s answer made sense to me. How it lands in AgentSpace is something I’ll find out over the next few sprints.