ThinkingBox evaluates agents by the final state of the system

On October 3, Microsoft and Hugging Face published a new overview of ThinkingBox, an environment and benchmark for evaluating agents that work on stateful business workflows. The accompanying paper, revised on October 1, describes 507 tasks across retail, auto insurance, travel, neobank and IT/HR support scenarios.

Each task starts from a defined initial state, exposes Model Context Protocol (MCP)-compatible tools and ends with executable checks over the backend state. The goal is not to decide whether the agent's answer sounds correct, but whether the database and resulting side effects actually match the requested outcome.

The authors use a support ticket for a delayed delivery as an example. The agent checks the order, tracking, customer profile and policy, opens a ticket and then closes it as resolved. The sequence looks orderly, but the carrier still has an open exception: the correct ticket state should have been "hold", not "solved".

ThinkingBox therefore treats the agent trajectory as a claim and the terminal state as evidence. For tasks with structured effects, deterministic judges compare what changed with the required end state and reject wrong values, missing actions or unintended extra changes.

The 507 tasks are repeated 20 times from a clean state. The benchmark separates average single-run success, whether a task can be solved at least once, and how many tasks are completed correctly in all 20 recorded attempts.

That distinction changes how the results read. In the report, Claude Opus 5.5 reaches 67.16% pass@1 but completes 241 of 507 tasks in all 20 runs, or 47.53%. Kimi-K3 solves 476 of 507 tasks at least once but only 68 in all 20 attempts. The benchmark is not only asking what a model can do; it measures how often the same workflow stays correct when repeated.

In a common-set analysis across 12 models, the authors report 121,680 valid trials. Of those, 79,853 fail the executable checks. Yet 67.24% of the failures still terminate cleanly, include at least one state-changing action and report no final tool error.

Within those failures, executable checks find wrong field values in 77.61%, unintended extra effects in 43.30% and missing required effects in 25.36%; the categories overlap. A monitor that only watches exceptions, formally valid tool calls or the agent's final text can therefore mark work as successful even when the system of record shows an incomplete or incorrect outcome.

The practical design lesson is to separate execution from verification. If an agent updates a CRM, ticketing system, order record or customer master, the outcome can be checked by reading the authoritative state after the action and comparing it with explicit conditions, rather than treating the model's final message as proof.

The authors also suggest classifying tool and system errors so retries target recoverable failures, reducing the tool surface to what the workflow needs, and keeping human approval for changes that are difficult to reverse. ThinkingBox makes these choices measurable because each attempt starts in isolation and exposes the effects it produced.

The 507 tasks model enterprise patterns, but the post states that every public workflow is a synthetic reconstruction and does not contain real customers. In addition, 477 tasks are graded on state alone, while 30 add a narrow binary response rubric where a requirement cannot be expressed as a simple database value.

The advantage is that the environment, data and checks are inspectable. The ThinkingBox framework is open source, the v1.0 benchmark is released separately, and its OpenEnv integration can run episodes and return a pass/fail result.

For a company, this does not replace evaluation on its own processes. It does provide a concrete criterion to bring into internal tests: define the correct final state first, then verify whether the agent reaches it repeatedly.