OhMyBugA code review benchmark built to make decisions, not headlines.
OMBench is the benchmark we run before we change how OhMyBug reviews code. Every bug in it is checked against the code. The cases that decide are held out from tuning. Every result comes with its price and its speed. Apply, and we may run your reviewer on it too.
In numbers
What it measures
A review is useful when it finds the bugs that are there, stays quiet when there are none, and does it at a price and speed you can live with. OMBench reports all of these side by side, so a gain in one is never bought silently with another.
Recall
Of the real bugs in a change, the share the review found. Reported for all bugs, for hard bugs that few reviewers catch, and for omissions, where the change forgot to touch another file.
Precision
Of what the review flagged, the share that is a real bug. Also reported raw: every medium-or-worse finding counts, with the number of findings per change.
F1
Recall and precision in one figure, averaged per change, with a 95% interval.
Noise on clean code
Findings per change on changes with no known bug, and the share of clean changes that got no finding at all.
Price
Real cost per change, read from the meter, not estimated. A run with an unmetered change is flagged, not filled in.
Speed and failures
Wall time per change (median, 90th percentile, worst), and how many runs failed or timed out.
Where the cases come from
Three kinds of real code changes, never synthetic ones:
Public review benchmarks
Pull requests from open-source projects with labelled bugs, re-checked against the code before use.
Replays of real fixes
The commit that introduced a bug later fixed upstream, including public security advisories. The review sees the change as it was, before anyone knew.
Bugs reviewers missed
A growing set of bugs that slipped past AI review, confirmed by the fix that followed.
The cases span many languages and frameworks and four sizes, from a few lines to changes that touch more than ten folders. Clean changes with no known bug are in the set on purpose: without them, a reviewer that flags everything looks good.
How each bug is verified
A benchmark is only as good as its answer key. Before a bug can count, a script checks that its location exists in the code at that commit and sits on lines the change touched or activated. A bug whose location is outside the change is audited by hand. If the change did not introduce it, it is left out of scoring (35 so far), without editing the source data. For an omission, where the fix lands in a file the change never touched, a finding is credited on either the fix site or the line in the change that triggered it (47 such anchors so far).
Four stages
A run moves through four stages in order. Each stage is small enough to be cheap and large enough to answer its own question.
| Stage | Cases (held out, v1) | The question it answers |
|---|---|---|
| S0 Smoke | 6 | Does it run on small, large and very wide changes without crashing, timing out or blowing the cost? |
| S1 Core | 22 | How good is it on changes where reviewers disagree, so the score can tell two reviewers apart? |
| S2 Breadth | 28 | Does it hold on wide multi-folder changes, on other frameworks, and on clean changes? |
| S3 Missed bugs | 16 | Does it catch real bugs that reviews missed before, at the commit that introduced them? |
Cases are picked by how well they separate reviewers: a bug that every reviewer catches, or none does, tells you nothing. Selection uses the changes; scoring then counts every verified bug in them, the hard ones included.
Scoring, by code
No hand arithmetic. A model judge matches each finding to the known bugs of its change and must point to the same defect at the same place. A script builds the report and refuses to print a verdict when any case is missing, unjudged, or was run on a different version than the rest.
Differences between two reviewers are paired per change and come with a bootstrap 95% interval. We measured how much the same reviewer moves between two identical runs: about 8 points of F1 per change. That noise sets the stage sizes, and the rules for calling a result better, worse or a tie are written down before a run starts.
Held out, versioned, rotated
The pool is split into a development set and a held-out set. Everything we ever tuned on lives in development; the rest splits 40 / 60 within each change size. Nobody reads held-out results to tune a prompt or a rule. Each version of the set is frozen and named by its content hash. Every 60 days a fifth of the held-out cases moves to development and is replaced by fresh cases from after the newest models' training cutoff.
What we never publish
- The held-out cases themselves, or anything that identifies them.
- The answer key: which bugs are in which change.
- Which models or internal techniques any reviewer uses, ours included.
Get your reviewer benchmarked
If you build an AI code reviewer, apply to have it run on OMBench.
- You apply with the form below and tell us how your tool takes a change: an API, a command line, or an app on a repository.
- We pick the tools to run from the applications and agree the setup with you. We pay for the run on our side; your tool's own usage stays on your account.
- We run it on the held-out stages, the same way we run our own reviewer, on the same pinned version of the set.
- You see the report first, privately: every metric above, with its interval, price and speed.
- You decide whether your result goes in the public table. Without your yes, it stays private.
Apply
We use your email only to reply about this application and delete the application after a year. Privacy. No account or card needed.