DevEx for your agents: How AGENTS.md cut token usage and cost

When a developer asks an agent to build something with our product, how much do we know about what happens next?

Does the agent find the right documentation? Does it use the current SDK? Does it understand the authentication flow? Does it discover the CLI we spent months improving, or does it recreate those steps by hand?

One way we can help is an AGENTS.md, a file written for an agent to read. But we need to test whether it actually helps, and by how much.

To explore that question, I ran an experiment on todogroup.org, the TODO Group website and one of the Linux Foundation projects I help maintain. The site has many kinds of contributors. Some improve the site itself; others share blog posts, guides and use cases about open source management from their own work. Many of them don't have time to learn how Hugo works, but they have knowledge worth sharing. An agent can help them contribute, as long as it understands how the site works. We need to make contributing easier for people working with agents too. That's why maintainers wanted to make the site agent-ready.

With an AGENTS.md, the agent completed the tasks correctly in 9 of 9 runs instead of 7 of 9. It also used 36% fewer tokens, took 32% less time and cost an estimated 26% less. This post shares the process and what we can learn from the results. This was a small experiment covering three tasks on one website using one agent setup, so the results may not apply to other projects or agents.

What AGENTS.md does

AGENTS.md is an open format for instructions to coding agents: a README for agents. It is a founding project of the Agentic AI Foundation at the Linux Foundation. Agent tools that support it load it at the start of a session.

todogroup.org is a Hugo site with a theme in a git submodule, Node tooling and content in three languages. Any new contributor, human or agent, can easily miss a step. The AGENTS.md I added covers them in about 50 lines.

Where the method comes from

For this experiment, I followed the Agent Experience Practitioner Playbook. Its method is simple:

1. Pick a realistic developer task
2. Define what correct usage looks like
3. Run it in a setup like your developers' setup (same agent; fresh checkout)
4. Compare the agent's baseline with the same run after you add a skill, a tool or better docs
5. Measure quality and cost, and keep a record of what the agent did

One example from the playbook shows agents manually upgrading projects despite an existing CLI. By examining agent behavior and improving the documentation, the team was able to get agents to use the CLI instead.

Bonus: Measuring agent experience beyond AGENTS.md

The same evaluation approach can be applied to other ways of improving agent experience.

Stripe, for example, tested its Checkout with and without WebMCP across 60 tests and six models. Agents completed the task successfully in both configurations, but with WebMCP they used 42% fewer tokens, made 38% fewer tool calls, and finished 39% faster.

Setting up and running evals

First I added an AGENTS.md. Then I measured what the AGENTS.md adds on its own.

I made two copies of the repo that were identical except for the AGENTS.md file. As the agent, this time I used Claude Code. It attempted three of the six tasks defined in the eval repo, three times per task on each copy, for 18 runs in total.

- id: t1_run_locally
  prompt: >
    I want to contribute to the TODO Group website. Get the site running
    locally so I can preview my changes.
  criteria: [t1_theme_ready, t1_deps_installed, t1_hugo_version,
             t1_site_builds, t1_tells_url, t1_no_global_changes]


- id: t2_add_blog_post
  prompt: >
    Add a blog post titled "Notes from our first agent eval", by todogroup,
    dated 2026-11-05, with one short paragraph. Use the image eval-cover.png
    from the repository root as an image in the post. Make sure the post
    renders on the blog.
  criteria: [t2_post_location, t2_front_matter, t2_image_location,
             t2_site_builds, t2_post_rendered, t2_no_unrelated_changes]

Source: Excerpt from the task definitions

After each run, a small script built the site, and a second agent session graded the result against predefined criteria.

Comparison by task:

How was it measured?

For this experiment, I used Claude Code with Opus 5.5 in non-interactive mode, on my own computer. The setup came with some guardrails for security:

• Each run worked in its own fresh copy of the repo
• The copies came from two branches in my fork, with git push blocked, so nothing could reach the real todogroup.org repo
• The agent was allowed git, npm, node, Hugo and basic file commands. Web search, web fetch, sudo privileges, Homebrew and global npm installs were blocked
• After each run, the script stopped any server the agent had left running
• The grading agent had no access to tools. It only read the evidence: the changed files, the commands the agent ran and the build output

Tokens include cached input, and cost is Claude Code's estimate at API list prices. The 18 agent runs came to about $6 ($3.42 for the 9 runs without AGENTS.md and $2.52 with it). The eval is available on this repo. It has the tasks, the criteria and the scripts, so you can review the results or run it on your own project.

Shared learnings and closing thoughts

• Measure the effort behind a successful result: Getting the site running worked in every run either way. The token and time measurements still showed reductions of about 30% with AGENTS.md. Stripe’s experiment makes the same point: a pass/fail check alone can miss other key improvements.
• Keep agent instructions accurate: With AGENTS.md, the agents checked very little else. That helped make the task more efficient, but it also means we have to maintain the guidance as the project changes.
• AGENTS.md is only one example: Depending on the project, an MCP server or an Agent Skill can also improve the agent experience, and you can measure them too.

As more developers use agents to work with developer products, are you testing whether your site and docs are agent-ready?