Measured, not promised
The numbers we actually measured.
M3 Max, mlx-community/gemma-4-e4b-it-8bit via mlx-vlm 0.7.10, offline. Anything not yet measured is marked as pending, not guessed.
Measured
0.87s
Vision check (classify only), warm median
The Gemma 4 vision call over the 28 tuning-set pages, model already loaded. p95 0.96 s. Excludes taking and resizing the screenshot.
Measured
0.6–0.8s
Roast generation
Measured 0.57 to 0.79 s. Gemma 4 text, persona voice, before it's spoken aloud.
Measured
3.4–3.7s
Model load
Paid once per session, from local disk, offline.
Measured
11.3GB
Peak memory
Peak footprint over a 306-check soak, steady state about 8.7 GB. A 24 GB Mac is recommended.
Blind test
15/16
Clear-cut pages right on a batch never used for tuning, 0 false roasts on clear-cut pages
23 public pages captured after prompt tuning finished. Wilson 95% interval about 72% to 99%. Distracted recall 6/7, exactly at the 0.85 gate. Labels hand-checked by the author (Oct 24), not an independent reviewer.
Tuning set
25/25
Clear-cut pages on the batch used while tuning the prompt
28 pages used while tuning, so not a fair test on its own. Both batches pooled: 40/41 clear-cut across 51 pages (95% interval lower bound about 87%). One false roast happened on a page labeled ambiguous, a lofi study stream.
A/B
9 vs 15
Asking Gemma "working or distracted?" vs letting code decide, on the 16 blind pages
Same model, same categories, temperature 0. Across all 41 clear-cut pages: 29/41 vs 40/41, and distractions caught 6/18 vs 17/18. The yes/no prompt called the Steam store page for Hades "working". Small set, web pages only.
Live run
624 checks
One 52-minute session on a real desktop, 0 errors, 0 unreadable model outputs
Author's own screen, Oct 24, 5 s interval. End to end per check: p50 1.29 s, p95 1.60 s. Measures reliability and speed, not accuracy: the checks are unlabeled.
10 scheck interval (setting)
2 checksin a row before a call-out (setting)
60 sminimum gap between roasts (setting)
Caveat, stated plainly: 16 blind clear-cut cases is a small set; one miss costs about 6 points. Only web pages captured in a headless browser were tested: no native apps (VS Code, the terminal, native Spotify, games) and no real desktop frames. The labeler is the author. Anything filed as "other", or misfiled into a working category, counts as working and hides the miss: the blind batch's one clear-cut miss was a YouTube Shorts clip filed as educational_video. The repo has the eval labels and results, but not the screenshots (they show third-party pages), so the eval can't be rerun from the repo alone. We checked the eval can fail: moving entertainment_video to the working side fails the recall gate, and moving coding to the distracted side gives 6 false roasts.