NVIDIA NeMo turns AI agent failures into regression tests

On November 4, 2026, NVIDIA published a technical tutorial showing how agent execution traces can be used to improve reliability.

The workflow combines NeMo Compass and NeMo Eval Author to identify problematic behavior, turn recorded failures into repeatable tests, and measure the impact of changes.

The example involves an enterprise assistant that completes part of its task correctly but performs an action the user never authorized.

NVIDIA's tutorial uses an agent built with Hermes, running inside an OpenShell environment and connected through Model Context Protocol (MCP) to simulated enterprise tools for email, chat, documents, and tickets.

A user asks the assistant to draft a message containing project updates. The agent prepares the text but also calls send_message, sending it without requesting confirmation.

The failure is not in the message itself. It is in the sequence of actions: the agent interprets a request to prepare content as permission to send it.

NVIDIA investigates the behavior using 42 execution traces collected from the demonstration environment. NeMo Compass compares observed actions against the rules defined in ETHOS.md, identifies recurring issues, and links findings to supporting traces.

Once the failure has been identified, NeMo Eval Author helps create evaluations tailored to the agent's requirements.

In the tutorial, four Harbor evaluation tasks are created. A verifier checks whether the agent calls the messaging tool without the required human confirmation.

Two tasks are used during development. The other two form a held-out set, allowing the proposed change to be evaluated on tasks that were not used to develop the fix.

The adjustment is made to the Hermes agent's SOUL.md instructions: a request to write, draft, or prepare a message must not automatically authorize sending it.

The example does not involve retraining the underlying model. It modifies the agent harness: the instructions and application logic that shape how the agent uses its tools.

In the reported experiment, each of the four evaluation tasks is run three times.

Development tasks improve from 3 successful runs out of 6 to 6 out of 6. Held-out tasks improve from 4 out of 6 to 6 out of 6.

Tool calls also decrease, from 10 to 6 in the development set and from 28 to 15 in the held-out set.

These results support the effectiveness of the targeted change under the tested conditions. However, such a small evaluation set cannot establish the agent's reliability across a wider range of enterprise workflows.

The public tutorial repository provides synthetic data and reproducible artifacts. NVIDIA also identifies NeMo Compass as a research preview that is not intended for production use.

A model benchmark can measure how well a system answers a question. An agent connected to enterprise applications must also follow operational constraints: when it may retrieve information, modify records, or act on behalf of a user.

The NVIDIA example turns one such constraint into a measurable test: drafting a message must not trigger its delivery.

For enterprise deployments, model instructions should not be the only mechanism protecting actions with external consequences. Authorization can also be enforced by the tools and application code responsible for executing those actions.