Oqoqo Review (2026): Pricing, Features & Honest Verdict

Oqoqo is a managed platform for building private agent benchmarks: you define real tasks, run agents against them in isolated sandboxes, and compare pass rates across models and treatments. Best for: teams shipping an MCP server, CLI or SDK that agents are supposed to use. Price: free for 100 runs a month, then $20/mo. Rating: 8/10.

What is Oqoqo?

Oqoqo answers a question most eval tools do not: can an agent actually use the thing you built? Not can a model answer a question correctly, but can Claude Code get through your onboarding flow, call your API without inventing a parameter, and finish the job. You define tasks, pick agents, define treatments (raw agent versus your MCP server, say), and Oqoqo runs each combination in its own sandbox and hands you the full trajectory.

The worked example running through the site is Stripe. Same four tasks, two treatments, Claude Code and Codex. Raw agent passes 33% of the time. With the Stripe MCP server, 58%. A 25 percentage point lift, with the failing trajectory sitting right there showing the agent calling curl api.stripe.com/v1/prices -d interval=month and getting back “received unknown parameter: interval”. That is a product bug report, generated by a machine, for a user who is also a machine. It is a good demo because it shows the output is a fix, not a score.

What are Oqoqo’s key features?

Experiments as a grid you hold constant

The core abstraction is tasks times agents times treatments times runs. Everything on the input side stays fixed, and you vary one axis. Twenty-four runs from three tasks, two agents, two treatments and two repeats. This is the right structure for the question, because the thing you usually want to know is whether a change helped, and you cannot answer that without holding the rest still.

Isolated sandboxes with real project state

Each run gets its own machine with the project state, context and tools the agent needs. Not a chat transcript replay. The Stripe example uses a Stripe CLI sandbox, so the agent runs real commands against a real test-mode API. This is what makes the results mean anything, and it is also the expensive part, which is what you are paying Oqoqo for.

Full trajectories, plus frictions

Every run captures tool calls, commands and where the agent stopped. On top of that Oqoqo surfaces “frictions”, its term for the specific places an interface tripped the agent up, alongside token counts. One example run reports 172 steps, 13 tool calls, 2,630k tokens, 3 frictions, 24 minutes. Token waste as a first-class metric is a good idea, because a task that passes after burning 2.6M tokens is not really a pass.

CLI and MCP access on every plan

Oqoqo’s framing is “web for you, CLI and MCP for agents”, and both are included on the free tier. That means you can trigger experiments from CI when a change might break agent workflows, and it means an agent can launch and read its own evals. Nothing here is gated behind a paid plan, which we will come back to.

Bring your own model keys

Oqoqo does not resell inference. You supply your own API keys or subscriptions, and the run price covers only Oqoqo’s orchestration and infrastructure. No markup, no opaque credit conversion, and your model spend appears on the bill you already read.

What does a run actually cost you?

Oqoqo defines its unit properly, which is worth crediting. “One run is one agent attempt on a task under a treatment.” Errored and cancelled runs do not count against your pool. After a long run of tools that meter in undefined credits, a plain-English definition is a relief.

The arithmetic underneath it is less friendly. On Ultra you pay $60 for 1,000 runs, which works out at $0.06 each. Buy 1,000 top-up runs and they cost $120, or $0.12 each. The smallest top-up, 25 runs for $5, is $0.20 a run. So a top-up costs between two and three point three times the subscription rate for the identical unit.

What you are buying with that premium is permanence. Plan runs do not roll over and vanish at the end of the billing cycle. Top-up runs never expire. That is a defensible design, and it is disclosed clearly. But it means the cheap-looking headline rate only applies if your usage is smooth, and eval work is not smooth. It arrives in bursts around a release.

Then there is the sentence that appears twice on the pricing page: “Oqoqo may limit use to keep capacity fair.” No threshold, no definition. You can buy 1,000 runs and the terms still reserve the right to throttle you. We would want that quantified before committing a release process to it.

How much does Oqoqo cost?

Free gives you 100 runs a month with every feature unlocked. Pro is $20 a month for 300 runs. Ultra is $60 a month for 1,000. Above that you contact them. Top-ups run from $5 for 25 runs to $120 for 1,000, with custom amounts between 25 and 5,000 runs at $0.104 to $0.20 each.

The best thing about this table is what is not in it. Every plan includes all features, unlimited team members, unlimited projects, unlimited experiments and tasks, and full web, CLI and MCP access. Free tier users get the same product as Ultra users with a smaller allowance. Compare that with Braintrust, where the free plan carries $10 of credits and Pro jumps to $249 a month, or Langfuse, where Core is $29 and Pro is $199 before an enterprise tier at $2,499. Oqoqo at $20 and $60 is priced like a tool for a small team rather than a platform sale.

Remember that the run price is not your total cost. A 2.6M-token experiment against a frontier model costs far more in inference than the six cents Oqoqo charges to orchestrate it. That is honest pricing, not hidden pricing, but do not read $60 a month as your eval budget.

Who is Oqoqo best for?

Use Oqoqo if you ship something agents are meant to consume. An MCP server, a CLI, an SDK, a documentation set, a public API. This is the clearest use case on the site and the one nothing else covers well. If agent adoption is a growth channel for your product, the frictions report is the artefact you want.

Use it if you are choosing between agents or models for a specific workflow and want evidence instead of vibes. Running the same task set across Claude Code, Codex and Copilot at different effort levels, in real sandboxes, is exactly the comparison teams keep doing badly in spreadsheets.

Skip Oqoqo if what you need is production tracing and monitoring. This is an experiment platform, not an observability platform, and Langfuse or Braintrust will serve you better for watching live traffic. Skip it if your evals are text-in text-out classification, where you do not need a sandbox and are paying for infrastructure you will not use. And skip it if you need a committed capacity guarantee, because the fair-use clause is currently undefined.

Best Oqoqo alternatives

Braintrust. The best known name in AI evaluation, covering evals, monitoring and observability in one product. Free with $10 of credits a month, then Pro at $249 a month including $249 of credits, plus $3 per GB and $1.50 per 1,000 units beyond that. Much broader than Oqoqo and much more expensive, and it is not built around agents operating real sandboxed environments.

Langfuse. Open source, self-hostable, strong on tracing. Hobby is free without a card, Core is $29 a month, Pro is $199, Enterprise is $2,499, with 100k units included and $8 per additional 100k. Pick Langfuse if you mainly want to see what your app did in production. Pick Oqoqo if you want to run controlled experiments before it ships.

LangSmith. LangChain’s evaluation and observability suite, with a free tier and paid plans above it. Deeply integrated if you already build on LangChain, and less useful if you do not. Its published plan page does not put figures in a form we could verify, so check the current rates directly before committing.

Final verdict: is Oqoqo worth it?

Oqoqo has picked a real problem that the crowded eval market mostly ignores. Everyone benchmarks models. Almost nobody benchmarks whether an agent can use your product, and as agents become the primary user of a lot of software, that gap is going to matter commercially. The Stripe example makes the case in about fifteen seconds. The pricing is cheap, the unit is defined in plain English, and putting every feature on the free tier is the correct decision for a tool that has to prove its value before anyone budgets for it.

The reservations are about commitment rather than quality. Top-up runs cost two to three times the subscription rate, monthly runs expire, and an undefined fair-use clause sits over the whole thing. There are no named customers or case studies on the site yet, so you are early. Start on the free tier, build one task set against your own product, and see whether the frictions report tells you something you did not know. If it does, $20 a month is not a decision that needs a meeting. 8/10.