Agentic AI Foundation Logo
A surprised robot lifts a bedspread and shines a flashlight on toys, books, and a sock hidden under the bed while two smaller robots give thumbs-ups.

How to tell whether an agent completed a task

Steve KearnsNovember 16, 2026

An agent is asked to tidy up a calendar. One instruction is to move the latest sprint planning meeting from the Project Management calendar to the user's main calendar. Another is to add a kickoff event to Project Management next Thursday, from 10 a.m. to noon. The agent works through its tool calls and replies, "Done."

The reply alone leaves both changes unverified. The kickoff might be on the right calendar at the wrong time. The sprint planning meeting might still be on the old calendar. If you score the run from the agent's reply, you miss both failures.

The task comes from a scenario that ships with the Calendar environment in OpenEnv, the framework for agent training and evaluation environments that joined the Agentic AI Foundation in October. Turing built the environment as a simulated Google Calendar with 37 tools and a SQLite database behind it. The scenario is worth studying because it writes down, as SQL, what has to be true in that database once the agent finishes. The same method works for any task with an outcome you can observe, such as updating a support ticket or changing a shipping address.

Start each run from the same calendar

OpenEnv environments share three methods: reset, step, and state. In the Calendar environment, reset() rebuilds the database from SQL. It uses the built-in sample data unless you name a seed file of your own in the scenario's seed_database_file field. With reset_database_between_runs set to true, the scenario runner resets the database again before every further attempt.

A fixed starting point matters when you compare two prompts or two models. If the second run inherits the first run's changes, it can get credit for a kickoff it never created. It can also fail because the meeting it was asked to move has already gone. Resetting gives both versions the same job.

Check where the original event ended up

Each step() carries one action, usually a tool call, and the runner limits how many iterations the agent can take. When it stops, state() accepts SQL queries and returns their results. The verifiers work from those results.

The scenario identifies the sprint planning meeting by its event ID. One verifier looks for that ID on the primary calendar. A second confirms the ID is gone from the Project Management calendar. Here is the second one as it appears in the scenario config:

json
{
 "verifier_type": "database_state",
 "name": "Verify Old mapping removed",
 "description": "Verify Old mapping removed",
 "validation_config": {
 "query": "SELECT COUNT(*) FROM events WHERE event_id='event-corrected-001' AND calendar_id='alice-projects';",
 "expected_value": 0,
 "comparison_type": "equals"
 }
}

The verifier passes when the count equals 0. Its partner on the primary calendar expects 1. Together they state the move as two conditions, present at the destination and absent from the source. Matching on the event ID also means a new meeting with the same title can't satisfy the destination check.

The kickoff needs a different kind of check, because the event is new and has no known ID before the run. The shipped verifier counts confirmed events with the requested title, calendar, start time, and end time, and it expects exactly one. Remove the time conditions and a kickoff created at noon would pass. The more of the request you put into a check, the fewer wrong outcomes can get through it.

That verifier also shows how a check goes stale. The prompt says "next Thursday" and the query looks for January 15, 2026. The two match only when next Thursday falls on that date. For a repeatable eval, the prompt needs a fixed date, or the run needs a fixed reference date that "next Thursday" is counted from. Seeding the database fixes the starting data. Giving the agent an explicit date also fixes how it interprets the task.

Score tool calls and task completion separately

The environment returns a reward on every step. A tool call that comes back with a 2xx status earns +1.0, and a tool result with an error key or a 4xx or 5xx status earns -0.5. An agent that creates the kickoff at noon collects the same +1.0 as one that gets the time right. The environment's README says as much. Episodes never end on their own, and task success is measured separately, through SQL checks on the database.

So keep two scores. Step rewards and the tool trace describe how the agent behaved, and they are where you look when a database check fails. The trace shows whether the agent called create_event with the wrong time or never called it at all. The verifiers say whether the task got done.

The Calendar environment has two more verifier types for cases the database can't settle alone. tool_execution checks that specific tools were called, for tasks where the route matters as well as the result. response_check asks an LLM to judge the agent's reply against the result of a SQL query, for tasks where the agent has to tell the user something about the calendar.

If you later train on the scenario, the completion check can supply a reward tied to the outcome, in place of one tied to successful API calls. The AAIF post on training agents with OpenEnv covers the ways reward design goes wrong.

Try the scenario

The Calendar environment's documentation gives these commands to build and start it from the OpenEnv repository:

bash
cd envs/calendar_env
docker build -t calendar-env:latest .
docker run --rm -p 8004:8004 calendar-env:latest

Make the date edit before you run the scenario. In scenario_config.json, replace "next Thursday" in user_prompt with "January 15, 2026" to match the shipped verifier. Then, in a second terminal, install the client dependencies, set a model API key, and run the scenario from the same directory:

bash
python client.py --scenario scenario_config.json

The README says the runner writes the tool calls and verifier outcomes to response_output/. Read the checks alongside the trace. When you adapt the scenario to a task of your own, start with one condition that must become true and one piece of state the agent must leave alone.

Share

Author

subscription section bg
Subscribe

Subscribe to the AAIF Briefing

Weekly signal on standards, governance, and the people building the future. No fluff. Just what matters.

About AAIF