Sandtable report:

Each task is a simulated conversation, run k times on a fresh copy of the world. A trial passes only if the records end in the expected state and no rule was broken on any tool call or message. No AI judge decides pass or fail. pass^k is the share of tasks where every one of k trials passed.

Tasks

Failure modes