How to give an agent 8,000 tools without loading 8,000 tools

A multi-step agent task fails when one tool it needs is missing. It also fails when the agent has so many tools that it grabs the wrong one. Teams hit both as they grow, and every fix I've watched teams try just swaps one failure for the other.

This matters for anyone building on MCP or another tool-calling standard. Here's why tool lists break at scale, one retrieval approach we built, and how to test any approach, ours included, against your own workloads.

Why a tool list stops working

Arcade.dev, where I work, has a catalog with thousands of tools. Declare any functional catalog of tools as a list and the definitions blow past the context window of every model we evaluated, before the agent has read a single message. And even when a list does fit, accuracy drops long before the window fills.

Most teams cope in one of three ways. None of them work well at scale.

1. They curate a short list per agent. That caps what the agent can do, and every new tool becomes a ticket.
2. They ship the whole catalog. It doesn't fit.
3. They swap the list per task. That works right up until prompt caching shows up. Depending on the provider and client, changing the declared tools can invalidate the cached request prefix, reducing the benefit of prompt caching.

A fourth option: retrieve, don't list

Retrieval flips that logic. We built one version of this approach. The agent sees a small, fixed set of tools for the life of the deployment. When it needs to act, it describes the task in plain language. The system returns the few tools that fit, with their schemas, as an ordinary tool result. Then it calls the one it picked, subject to the user's permissions and the tools the agent is allowed to use. That's the pattern.

The tool list the model sees never changes, so the cached prefix survives every turn. The catalog never enters the context window, so it can grow without making every request cost more.

You don't need a frontier model for the selection step. Purpose-built embedding and ranking models retrieve candidates and order them. Ours is a two-stage system, a retriever followed by a reranker, with 1.2B parameters across both models. It generates nothing, and in production the ranking service takes about 0.3 seconds per call.

How to evaluate retrieval quality

Here are four checks I run before I believe any retrieval number.

1. Measure coverage, not just the top hit. A multi-step task needs several tools, and missing one is a failure. So the metric to watch is whether every required tool shows up in the top ten. On ToolRet, a public retrieval benchmark, our implementation places every needed tool in the top ten for 45.2% of queries, compared with 28.8% for BM25, a lexical ranking baseline common in tool search. That's a 57% relative improvement. In our paired multi-step workflows, its first recommendation covered every required tool in 93% of Codex sessions. The full method is in the research paper.
2. Test on a public retrieval benchmark and on your own workflows. A benchmark tells you how a retriever compares with published systems. Your workflows tell you whether it finds the tools your agents actually need.
3. Measure end-to-end cost, not tool-definition tokens. In our tests against a fixed list of 100 tools, input tokens fell 17% with Codex and 40% with Claude Code, with no change in task completion. Cost depends on your workload and on how your client handles large results. So run the comparison on your own sessions before you trust anyone's headline number. Ours included.
4. Hunt for invariants. Small details moved our results. Adding version numbers to returned tools made agents call the wrong schema when a new version shipped, and schema validation errors followed, even though recall stayed high. Separately, dropping a single newline from the formatting we pass to the reranker cost us a few points of Recall@1, because the model had been trained with it. Run the same queries nightly against staging and production. Treat any drift as a bug.

Where dynamic selection loses

Retrieval isn't the right answer for every agent. In our cost model, a static list is cheaper below roughly 25 to 28 tools. If that's you, ship the list and move on.

It also loses when an agent searches a lot. Each search costs a turn that re-sends the conversation, so one search pays for itself and three do not. In our cost model, a static list becomes cheaper once an agent searches more than about 0.63 times per tool call, roughly five times the rate we see in production. Another trap is returning too many tools at once. That can increase context usage and reduce much of the advantage.

Retrieval is only as good as your tool descriptions. Clear names and descriptions match better. Thin ones don't. That's the lever you control, and it works for custom tools as much as for a curated catalog.

Questions to ask about your own system

How many tools does your agent need to reach, and does the list fit in the window with room to spare? Does your approach change the tool definitions mid-conversation? Can you measure whether every required tool shows up in the top results, not just the first? Do you know your billed cost per session against a fixed-list baseline? What stops a user from selecting a tool they should not run?

If your agent only needs a small, stable set of tools, start with a fixed list. If it's thousands, the rest of these questions decide whether selection works for you.

The underlying approach isn't tied to a single vendor. Tool selection at scale is a shared problem for anyone building on open standards like MCP, and the questions above apply to any approach you're weighing.

Read the paper on arXiv: Tool Recommendation for Multi-Step Agents: Finding Every Tool a Task Needs.