OhMyBug
← OhMyBug/OMBench/Results
OMBench results

Each mode, measured. And measured again when its setup changes.

How many known bugs each OhMyBug mode finds, against how long a review takes. Every point is a stored run of OMBench, our held-out benchmark, with its date; when a mode's setup changes, its earlier points stay on the chart.

The full set ↓How to read it ↓

The full held-out set

All four modes on the full held-out set, as the modes page prints them today, and the figures it printed before, and an estimate.

The full held-out set: 85 changes, 742 known bugs. Each mode's solid point is what the site prints today; the hollow ones are what it printed before, or an estimate.
40%50%60%70%80%90%100%0 min1 min2 min3 min4 min5 min6 mintypical server timeknown bugs foundlite $1snap $2fast $4deep $12Nov 12Nov 15Nov 16Nov 12Nov 12estimate, Nov 17
40%50%60%70%80%90%100%0 min2 min4 min6 mintypical server timeknown bugs foundlite $1snap $2fast $4deep $12Nov 12Nov 15Nov 16Nov 12Nov 12estimate, Nov 17

what the site prints todayindicativeestimatethe same mode over time

The figures the site prints today, full held-out set, November 2026
ModePriceFound, of the known bugsHard cases foundPrecisionFindings per change: medium+ / allServer time: typical / 9 of 10Client wait: typical / 9 of 10
lite$161.5%52.5%79.9%30 / 4985 s / 2.1 min2.1 min / 2.6 min
snap$257.7%47.5%82.7%27 / 4534 s / 49 s56 s / 83 s
fast$476.5%77.5%82.5%35 / 611.9 min / 5.4 min2.3 min / 6.3 min
deep$12 / $2486.1%77.5%85.0%40 / 804.6 min / 10.0 min4.9 min / 10.7 min

Lite, snap and fast come from later runs. Hard cases: 40 bugs that earlier reviews missed, in 23 of the changes.

Earlier figures the site printed for the full held-out set, and an estimate
ModeMeasuredSetupFound, of the known bugsPrecisionFindings per change: medium+ / allServer time: typical / 9 of 10Client wait: typical / 9 of 10Price today
liteNov 16, 2026earlier setupindicative: measured on an older server version53.4%84.1%15 / 2484 s / 2.1 min2.1 min / 2.6 min$1
liteNov 15, 2026earlier setupindicative: measured on an older server version; scored with another judge setup; measured on fewer of the changes56.6%82.0%16 / 251.6 min / 5.7 min2.0 min / 6.1 min$1
liteNov 12, 2026earlier setupindicative: measured on an older server version; scored with another judge setup58.0%81.8%18 / 292.3 min / 5.6 min2.6 min / 6.0 min$1
snapNov 12, 2026earlier setupindicative: measured on an older server version; scored with another judge setup46.4%84.8%11 / 1931 s / 40 s45 s / 58 s$2
fastNov 12, 2026earlier setupindicative: measured on an older server version; scored with another judge setup72.4%85.5%18 / 342.0 min / 6.6 min2.3 min / 7.0 min$4
deepNov 17, 2026estimate87.5%79.8%56 / 1044.6 min / 10.0 min4.9 min / 10.7 min$12

How to read it

  • Found. The share of the known bugs a review found, all bugs of the set pooled. Known bugs are checked against the code before the run, in changes never used to tune any mode; a bug counts as found only when a finding points to the same defect at the same place. Nobody knows all the bugs in the code, so this is a share of the known ones.
  • Precision. The share of a review's findings that point to a real bug, averaged over the changes.
  • Findings per change. Your agent reads them all, checks each against the code and drops the false ones. The medium-or-worse count is what a fix starts from.
  • Server time. The chart's time, and the same measure as the modes page: from the moment the review starts on our side to the result. Typical is the median change; 9 of 10 is the time 9 changes in 10 stay under.
  • Client wait. What the person or agent who asked actually waits: from the start of sending the change to the answer. It adds packing and sending the change to the server time.
  • Earlier setup. One a mode ran before; each mode's solid point is the figure the site prints today.
  • Indicative. A point that shows the direction, not a like-for-like comparison with the solid point; the table says why.
  • Hard cases. Bugs that earlier reviews missed, a small sample.
  • Price. The price of one review in each mode today. When a review is charged is in the FAQ.

How the benchmark is built, verified and scored: OMBench. Which mode to run when: the modes page.