How to Run a 30-Day Agent Bake-Off (and Know What Each Agent Cost)
Picture a team that has been using AI agents for six months. Someone set up a coding agent. Someone else set up one for tickets. A third answers questions about the docs in a chat. The model bill arrives as one number, and when the CEO asks which agent drives it, nobody can say.
That is an easy spot to end up in, and it has an old fix: run a trial, measure each agent the same way, and decide with numbers. Here is how to run a 30-day bake-off for agents, which numbers Agent Barn gives you out of the box, and which ones you have to work out yourself.
Give every agent its own identity first
You cannot split a bill that arrives as one line. If several agents share one API key and nothing labels their calls, the log tells you what the key spent, not which agent spent it. You can add labels to shared keys, but then you are maintaining the labeling yourself.
The simpler route is one key per agent. In Agent Barn, each agent gets its own virtual key on a LiteLLM proxy that runs in your own Kubernetes namespace. A scheduled job imports the proxy’s spend log and attributes each call to an agent and an organization by matching the key. Reports read the stored cost records, and records keep the agent’s name after the agent is deleted.
Pick the agents and the question
A bake-off compares agents against each other, or an agent against the way the work gets done today. Pick two to five agents. More than that and you will not look at the results.
Write the question down before you start. Good ones:
- Which of these agents do people actually use?
- What does each one cost per week, and is that number steady or growing?
- Does a cheaper model do the same job on the same tasks?
“Are agents good?” is not a question you can answer in 30 days.
What you get without any work
These come straight from the Agent Barn costs reports, per the docs:
- Total spend, number of model calls, active agents, average cost per call and average prompt tokens, over 7, 30 or 90 days or a custom range.
- Spend over time, split by agent.
- On each agent’s page, a Costs tab: spend over time, token distributions, a breakdown by model, and a filterable list of model calls.
- Monthly totals, and for the current month a projection from month-to-date spend. The projection only appears once a day of the month has passed, because extrapolating from the first hour of the 1st would be noise.
Each agent also keeps a conversation history and a tool-call audit log.
What you work out yourself
Some of the numbers you want are not a report. You can get them with a bit of arithmetic, and you should be honest about which is which.
Conversations handled and tool calls made come from the activity records. Agent Barn does not put them next to the cost figures, so you line them up in a spreadsheet: one row per agent per week, with spend from the costs report and counts from the activity log.
Cost per task takes more care. The docs say cost reports do not allocate charges to individual conversations or tool calls. So you cannot read “this ticket cost $0.04” off a screen. Two workable methods:
- Matched tasks. Give each agent, or each model, the same ten tasks in a short window when nothing else is running, and read the spend for that window and that agent. This is an estimate: records land after a sync, and the window has to be long enough to catch them.
- Averages. Divide an agent’s weekly spend by the number of tasks it completed that week. This hides tasks that cost much more than others, so look at the cost-per-call distribution before you trust the average.
The reports also have limits you should know about. They are not provider invoices. They reflect the latest completed synchronization, so recent calls can take a while to appear and past totals can go up when missing charges are recovered. A missing record is not proof that a call was free. Use your provider’s billing for reconciliation.
Measure quality separately
No cost report tells you whether the work was good. You have to decide what good means for each agent and check it.
Pick one outcome per agent that you can verify: tickets created with the right fields, pull requests that needed no rework, questions answered without a follow-up. Check a sample by hand. Automated evaluations are an option too, for outputs you can score by rule or with a second model, but a person should look at least at the first batch.
Be careful with small samples. If you check 20 items and 17 are right, that is 85 percent, but the real rate could be well above or below that. Do not divide total spend by 17 and call it cost per correct result. Use the sample to compare agents that differ by a lot, and use the matched-task method above when two agents are close.
Set a ceiling before week one
A trial with no cap invites a surprise. Agent Barn supports a monthly spend limit for the organization and for each agent. Owners and admins set them. An agent’s effective limit is the lower of its own limit (or the organization’s default for agents) and the organization limit.
The limits are enforced through LiteLLM, using observed billing rather than reservations. Concurrent calls and late billing can push spend past the limit before it takes effect, so treat a limit as a ceiling with some give, not an exact cutoff. Pick a number you would be comfortable paying for a failed experiment. If an agent hits its limit in week two, that is a result too.
The 30 days
Week 1: run the agents on real work, not toy prompts. Open the costs page once, only to confirm that every agent shows up under its own name.
Weeks 2 and 3: leave them alone. On a fixed day each week, write down spend by agent from the report and the counts from the activity log. Note any agent that is idle. An agent nobody talks to is the easiest finding in the whole exercise.
Week 4: pull the numbers and run your quality check. For each agent, write one line: weekly spend, calls, tool calls, models used, and the checked sample result. Then decide: keep it, change the model, narrow its job, or turn it off.
If two agents cost about the same and one uses a smaller model, rerun the same ten tasks through both and compare the outputs. Model choice is the cheapest lever you have. In Agent Barn it is a per-agent setting within the organization’s default and allowlist. Changing it can change the agent’s behavior, so run your quality check again afterwards.
What a team said about wanting this
GG Group Media, a Lithuanian digital marketing agency, runs four Agent Barn agents in its development workflow. Before that, its agents each lived on their own, with no shared view of what they cost or what they were allowed to do. Paulius Pazdrazdys described what the team wanted:
“We wanted to solve the problem of managing the agents from one place and making them work in our stack, get cost control of those agents, and make them safe to use for the team.”
That is the bake-off in one sentence: one place, one cost view, agents the whole team can use.
What to do this week
- List every agent your team runs and who set it up.
- Check whether each has its own key. If some share one, fix that first.
- Pick two to five agents and one checkable outcome for each.
- Set a monthly limit per agent.
- Put the weekly check on the calendar.
After 30 days you will know what each agent costs, which ones get used, and which ones earn their keep.
Agent Barn is open source and runs on your own Kubernetes cluster. The cost and activity docs are at agentbarn.dev/guides, and the code is at github.com/aai-labs/agent-barn.