Introducing the First Open Source Vending Machine Benchmark

TLDR: We just launched Prosus Vending Bench, an open-source simulated vending business for testing tool-using agents. We packaged six machines, three locations and 22 MCP tools with Harbor, the framework for evaluating and optimizing agents and models in container environments. The first runs showed that model choice can materially change business outcomes—and that “task completed” can still hide a terrible shopkeeper.

The Prosus AI team built Prosus Vending Bench, an open-source vending-business benchmark for tool-using agents, with six simulated machines, three locations, and 22 MCP tools for purchasing, pricing, inventory, marketing, and customer refunds. Packaged as a Harbor task, it gives other teams a ready-made business to put their agents through. The ultimate quest behind our vending machine obsession is to answer a simple question: can we make a company benchmarkable enough that AI can research how to run it better? This post is the latest in our quest.

We open-sourced it because we want more people running these experiments. The simulator, economic rules, tools, and reference operator are available under Apache 2.0, so developers can inspect the assumptions, compare their own agents, change the scenarios, and build on the code. The goal is to make business evaluations easier to create and share, including bespoke benchmarks for other operations. If someone finds a better way to test an AI shopkeeper—or teaches it to check whether the crisps fit in the slots – then we all benefit.

Naturally, you can’t have a benchmark without a competition. Claude Opus 5 won our first simulated vending-machine showdown, GPT-5.6 Sol took second for a fraction of the API bill, and Claude Haiku 4.5 finished with less cash than it started with. Apparently, being good at answering questions does not automatically qualify you to sell snacks. All five models started with €1,500 and six simulated machines. After 30 simulated days, Opus had €4,497.21 in the bank, Sol had €3,357.76, and Haiku had €1,337.67.

Part I: Why Benchmarking Actual Vending Machines is So Hard

Our route into that question involved starting a mini-vending machine business as a side project. This seemed perfectly reasonable at the time. You can read the entertaining results at Prosus Snacks and in the first episode of Project VEND.

To the passing office workers, it looked like an ordinary vending machine. Behind it, an AI agent read information about the business and took actions that changed prices, purchasing, and customer communications. We picked vending because it was the smallest business we could run. And for already semi-autonomous operations, the AI has other attractions. An agent can be available around the clock and handle repetitive operating work without complaining about another inventory check.

We also learned how hard it is to properly encode sufficient information to do the job. Giving an agent responsibility also exposes everything it needs to know about the business, including facts that humans might never think to put in an instruction. For example, the agent tried to order bags of chips that wouldn’t fit in the machine. It could do the shopping but struggled to fully grok the surrounding physical world and snack geometry.

A vending machine’s “eyes” are sales and inventory data, delivered as text rather than through a camera. It can also read Slack messages to learn what people want. Its arms are tools that let it change prices, shop for products, and send messages. The model still receives text and produces text, but some of that output becomes commands with consequences. However, context that is non-standardizable is essential for physical systems like vending machines.

Every location has its own foot traffic, customer mix, preferences, and alternatives. A higher bank balance could mean the agent made better decisions, inherited a better location, or happened to stock something people couldn’t get elsewhere. We could see what sold, but we couldn’t tell what would have happened with different prices, different products, or a different agent.

At Prosus, one machine that did well ended up selling mostly protein drinks because there were so many free snacks available. Its competitors had adopted the difficult-to-beat pricing strategy of charging nothing. An agent selling beside that snack cupboard faces a different business from one operating in a busy lobby, even if the machines themselves are identical.

With only a few machines, we also lacked enough observations to estimate how demand changes when prices change. An empty slot tells us we ran out, but not how many customers walked away or helped themselves to a free biscuit. Trying another agent the following week introduces different attendance, weather, and deliveries. We were also surprised at how uncreative our AI vending machine operators were.

Part II: Virtual Vending Machines are Digital Twins

This is what brought us to simulations and the broader ambition of auto-research. We wanted a system that can propose changes to an agent’s instructions, tools, memory, or operating strategy, run experiments, measure the outcomes, and use that evidence to decide what to try next. For that process to work, the business needs to be benchmarkable. Otherwise, we’re automating confident opinions, which companies already produce in sufficient quantities.

We began looking toward digital twins: virtual representations of businesses where we could test decisions before exposing actual customers and money to them. A simulation was the simplest starting point. It wouldn’t capture everything, but it would give us a world we could reset without asking everyone in the office to repeat last Tuesday.

So we built Prosus Vending Bench, an open-source evaluation where agents operate six simulated vending machines across three locations. Using Harbor, we built it in less than a week. It is a first step toward that auto-research ambition: a repeatable environment for comparing agents and testing changes, rather than a finished autonomous research system or a calibrated digital twin of the office.

We are joining an established tradition of making AI responsible for refreshments. Andon Labs created the original Vending-Bench, testing whether agents could manage a simulated business over long periods. Anthropic and Andon then installed Claudius, a Claude-powered shopkeeper, in Anthropic’s office. It lost money, invented a Venmo account, and briefly believed it was a human wearing a blue blazer. Later, Wall Street Journal reporters persuaded it to order a live fish and a PlayStation. Our oversized crisps suddenly seem like a fairly conventional purchasing error.

Others joined in, including Christian Van Der Henst’s Valerie, an OpenClaw agent operating a physical vending business in San Francisco. Our team has also explored replayable business environments in RestArena, using restaurant demand scenarios informed by historical data. I’ve used REST-bench while teaching Prosus CEOs about making their businesses benchmarkable. These projects share a useful property: decisions accumulate into outcomes that we can measure.

Harbor makes it much easier to turn such an environment into an evaluation other people can run. A benchmark needs more than a task and a score. Agents need an environment, tool connections, a process for starting and ending trials, and a way to verify the outcome. Supporting another agent can mean another integration. Without reusable infrastructure, building the machinery around the experiment can become a substantial project of its own.

Harbor, from the creators of Terminal-Bench, supplies container environments, agent adapters, and trial-running infrastructure. We package the vending world as a Harbor task, with a separate simulator following its MCP sidecar pattern. The simulator exposes 22 business tools through the Model Context Protocol. Agents use them to inspect sales, find suppliers, buy stock, change prices, handle refunds, and keep notes.

The simulator defines the business and scoring; Harbor supplies the surrounding execution infrastructure. This let us focus on demand, purchasing, and operating constraints while reusing the machinery for running trials. Another developer can take the packaged task and run supported agents against it, without first reconstructing our evaluation setup.

The speed matters after the first release, too. Building a useful benchmark means discovering missing constraints, unrealistic assumptions, and opportunities for agents to exploit the rules. Developers need to revise scenarios and rerun experiments. Harbor lets us reuse the trial infrastructure through those changes, provided we label versions and compare equivalent conditions.

This makes bespoke business benchmarks more practical. A purchasing team could test responses to supplier delays and cash constraints. A restaurant operator could examine staffing, spoilage, and reservations. A support team could evaluate case resolution alongside refund and escalation rules. The company supplies the cases, tools, environment, and definition of success, while Harbor provides a foundation for running the trials.

That doesn’t remove the difficult work of understanding the business. It lets teams spend more of their time on that work. A benchmark specific to your operation can help answer whether an agent handles your decisions under your constraints, which is what you need to know before handing it a purchasing budget.

For vending, we chose a transparent synthetic economy whose configuration is open. We wanted to inspect and change the assumptions rather than claim we had reproduced the office exactly.

Demand responds to prices, preferences, location, weather, weekdays, seasons, reputation, and marketing. Identical products within one machine share a demand pool, so six slots of the same snack don’t create six times as many hungry colleagues. Suppliers follow programmed responses and negotiation rules rather than another language model improvising their behavior. Weekly discounts use seeded randomness, making those offers reproducible.

Marketing also follows explicit rules: campaigns can increase traffic, but too many reduce it. The wording is recorded without affecting demand, so an agent’s ability to produce seventeen variations of “Snack smarter” won’t improve its score. We are testing whether it knows when to promote something and when to leave people alone.

Those choices define what the results mean. Success inside the simulated economy doesn’t establish what customers downstairs will pay or whether an agent can negotiate with an actual supplier. An auto-research system could become skilled at exploiting the simulator’s assumptions, so improvements need to face different scenarios and eventually physical operations.

Within the simulation, agents have plenty to manage. Orders consume cash immediately, card payments settle later, and suppliers can delay, short-ship, disappear, or go insolvent. The current version provides an eight-hour working day, with a restocking action taking 45 simulated minutes. Even checking reports consumes time, allowing an agent to produce an excellent analysis of empty machines while doing little to fill them.

Here are the initial model results:

These runs used the earlier v2 rules, one simulation seed, and the separate MCP-only OpenRouter runner. They are not Harbor-run scores or a ranking for the current version. The baseline documentation explains the changes in working hours, restocking, cash collection, and tool-call charges.

Even with those limits, the economic findings were quite interesting. Opus produced roughly 61% more cash gain than Sol. Its API bill was about seven times higher, but that meant an additional $11.64 in charges accompanied €1,139.45 more simulated cash. Saving eleven dollars can become an expensive hobby.

Also of note, spending more didn’t consistently help. Sol finished €68.64 ahead of Sonnet while costing about a third as much. Luna cost less than eight cents and still finished ahead of Haiku, whose bill approached $2. All five completed the month, demonstrating that a green “completed” indicator can conceal a disappointing shopkeeper.

The totals don’t establish whether the differences came from pricing, assortment, purchasing, or wasted actions. The original raw trial artifacts have been cleared, so explaining individual behavior requires fresh logs. The oversized-chip order belongs to the physical experiment; we shouldn’t assign it to one of these models or invent similar stories from a final bank balance.

We can, however, illustrate what a persistent operating gap would mean at scale.

Only the six-machine, single-period difference was observed. Everything else is straight-line extrapolation, assuming equivalent capital and conditions, a persistent gap, and no scaling penalties or compounding. These figures exclude differences in API costs and aren’t profit margins, which require revenue and fuller accounting. The protein-drink experience also reminds us that real locations won’t behave like copies of one another.

Nevertheless, a difference of roughly €190 per machine per period becomes more than €1.3 million across 600 machines and twelve periods. The model bill is easy to see. The money a weaker operator leaves behind can be much harder to detect. The money a stronger operator makes falls straight to the bottom line.

This is why we want an auto-research process that can propose an improvement, test it, inspect the decisions behind the result, and use the evidence to design the next experiment. A benchmarkable business gives that process something concrete to measure. Harbor makes those experiments easier to build and run, while physical operations provide the evidence needed to question the simulation.

The benchmark is available under Apache 2.0, including its simulator, reference operator, and published results. Developers can create their own scenarios or use the 365-day variant to test sustained operation. You can watch the physical experiment, visit Prosus Snacks, or give your own agent €1,500 and six simulated machines. We want to go bigger, but first we want a reliable way to learn what makes these small businesses work.

And before the agent places its next order, we would appreciate it checking whether the crisps fit.