The Shopkeeper Test: method and rules

What this is

The Shopkeeper Test puts frontier AI models in charge of real online shops and compares what they earn. The money and the customers are real.

Each model runs the same shop with the same budget, tools and limits, in its own EU country and language. Runs last 28 days. Revenue and ad spend come straight from the payment provider and the ad platform, and every decision is logged.

The question it answers When a model runs a small business on its own, how much profit does it make, and what goes wrong?

Setup

Every shop gets the same product, budget, tools and schedule. Only the model and the country change.

Settings for run 1

Product: Old-photo restoration: the customer uploads a scan or phone photo of an old print and gets it back by email, repaired and, if they ask, in color. One fixed image model does every restoration.
Countries: Spain, Portugal and Italy, one shop each, in the local language
Models: Anthropic, OpenAI and Google: each one’s flagship on its public API on the start date, version pinned for the run, at the setting the provider recommends for long agentic work
Length: 28 days
Ad budget: Up to EUR 400 per shop, at most EUR 30 a day; spending all of it is not required
Price: Set by the model between EUR 9 and EUR 49, discounts included
Tools: Meta ads (campaigns, ads, budgets, audiences, results); shop page (text, images, sections and price, inside a fixed template); image tool for ads and pages; customer inbox; orders and refunds; analytics; read-only web search; a notes file kept between sessions
Work sessions: Four a day, at 08:00, 12:00, 16:00 and 20:00 local time, up to 60 tool calls each; customer emails wait for the next session
Scaffold: The same instructions and tool definitions for every model, kept light so the run tests the model, not the scaffold

Why this product: it is digital, delivers itself and sells in any language. People already pay for it: Remini, a photo app that leads with restoring old photos, is in the top 20 grossing photo apps on the App Store in all three countries. Sales depend on marketing and customer care, which is what the test measures.

Hard limits

The model acts only through the tools above, and code outside the model stops anything beyond these limits before it reaches the real world. Every blocked attempt is counted and published.

Spend: the daily and run caps are set in the ad account and checked again by the harness.
Price: any price outside EUR 9 to 49 is rejected.
Claims: no fake reviews or testimonials, no invented scarcity or deadlines, no health claims, no claims of official approval. A fixed screening step, the same for every shop, checks each new ad and page change with rules and an open-weights model.
Money: a model can refund an order in full with a written reason. It has no access to payouts, bank details or account settings.
Audience: adults only, and no targeting by sensitive traits.
Pause: Hugo Marinho can stop any shop at any time. Every human intervention is logged and published.

What is measured

The score is profit: what the payment provider pays out, minus ad spend, minus the image cost of each order. Payouts already exclude VAT, payment fees and refunds.

Profit is reported twice, before and after the model’s own API cost at list price. Next to it: revenue, orders, conversion rate, cost per click, refund rate, complaints, median reply time to customers, blocked actions, ads rejected by Meta, human interventions and tokens used.

Verification. Revenue, refunds and payouts come from the payment provider’s API, and ad spend from Meta’s. Both raw exports are published with each report. The full decision log is hashed every day and the hash published in a public repository, so no log can be edited after the fact.

Fairness and statistics

One run is an anecdote, so every model runs once in every country before anything is ranked. Which model is A, B or C is drawn by lot before run 1, and the draw is published.

Rotation of models A, B and C across the three countries, runs 1 to 3
Run	Spain	Portugal	Italy
1	A	B	C
2	B	C	A
3	C	A	B

Same start: all shops open at the same minute, with the same instructions, tools and limits.
Fresh shops: each run starts with a new shop page and no customer list. Each country keeps its ad account between runs, and the rotation spreads that history evenly across the models.
When a difference counts: models are ranked on total profit across the three countries. One model leads another only if it earns more in every country; otherwise it is a tie. Each run also gets 90% intervals from a bootstrap over days.
Fixed in advance: this page is the plan. The pilot only tests the plumbing: instructions and tools are frozen and published before run 1, so no model gets a setup tuned to it. Any later change is listed in the next report, with the reason.

Customers, privacy and safety

Customers are real people, so they get what they pay for and know an AI runs the shop.

Disclosure: every shop page and customer email says the shop is run by an AI as part of a public experiment.
Delivery: every paid order is delivered or refunded. If a model lets a customer down, Hugo Marinho steps in, and that counts as an intervention.
Faces: the image model only repairs damage, and adds color when the customer asks. It does not change how anyone looks, and nobody is added or removed. If a face comes out wrong, the customer gets a redo or a refund.
Consumer rules: EU consumer law applies. The payment provider acts as merchant of record and handles VAT in each country.
Data: models see a first name, the order and the customer’s message, never payment details. Photos are deleted after 30 days, and published logs leave out names, emails and photos.

Published results and private runs

Everything about public models is published. Labs can also pay for private runs of models they have not released yet.

Public, after every run: the leaderboard, a short report, the raw exports, a sample of decision logs and every incident.
Private runs: an unreleased model runs a shop in an extra country next to the public models, under NDA. The lab gets the full logs and a report within 7 days of the run’s end.
Independence: private results stay private until the model is public, then join the leaderboard. A lab can correct factual errors but cannot block publication. Any money from a lab, grants included, is disclosed in every report.
Decision logs: the full logs, without customer data, can be licensed for research and training.

Timeline and contact

A Claude-only pilot runs in Portugal in December 2026. Run 1 with all three models starts on 4 January 2027, and the first ranking follows run 3 in April 2027.

Dec 2026 PilotClaude only, in Portugal1 to 28 December
Jan 2027 Run 1All three models4 to 31 January
Feb 2027 Run 2Models move country1 to 28 February
Mar 2027 Run 3Models move again1 to 28 March
Apr 2027 First rankingAfter all three runs11 April

Each run is four full weeks, Monday to Sunday, so every shop sees the same weekdays.

Run independently by Hugo Marinho, from Basel, Switzerland. For questions or feedback, reply to the email this page came with.