I benchmarked Ox Alpha on SWE-bench Verified Mini (50 tasks): 96% resolved. Now I’m skeptical of myself.

TL;DR: The result looks TOO GOOD TO BE TRUE — that's exactly why I'm posting it. I ran Ox alpha on the entire SWE-bench Verified-Mini set with the official mini-swe-agent scaffold — the same agent used for the swebench.com Bash-Only leaderboard. Result: 48/50 = 96% resolved, judged locally with the official SWE-bench Docker harness. For scale: Claude Fable 5 scores 95% with Anthropic's own tuned agent scaffold, and under my identical bash-only scaffold the best models score ~77% on the full 500-task set. A free-tier model landing above Fable 5 should trigger your BS detector — it triggered mine. Read the caveats below; there are good reasons my number may be inflated. —— SETUP —— • Model: Ox alpha, served through opencode's Go gateway • Scaffold: mini-swe-agent v2.4.6 (Bash-Only official leaderboard scaffold) • Prompt/config: Official swebench.yaml tool-call template, unchanged except pointing at the model; step limit 250 • Dataset: swe-bench-verified-mini — the well-known 50-task subset of SWE-bench Verified huggingface.co/datasets/MariusHobbhahn/swe-bench-verified-mini huggingface.co/datasets/princeton-nlp/SWE-bench_Verified • Judging: Official swebench harness run locally; resolved only if ALL FAIL_TO_PASS and PASS_TO_PASS tests pass • Hardware: One Windows 11 box, Docker Desktop (WSL2), one container per instance —— RESULTS —— • Resolved: 48 / 50 = 96.0% • Failures: django__django-11790, django__django-11815 • Empty patches / errors: 0 / 0 (all 50 submitted a real patch) • Avg steps: 40 (max 116) • Total wall clock: about 2h04m with 4 parallel workers • Per repo: django 23/25, sphinx-doc 25/25 —— HOW THIS COMPARES TO OTHER MODELS ON SWE-BENCH VERIFIED —— Two kinds of numbers exist: the official Bash-Only leaderboard (swebench.com) runs every model through the identical mini-swe-agent scaffold (directly comparable to my setup), while vendors also self-report scores using their own tuned agent scaffolds. This run: ox-alpha-free (mini-50 subset, only 2 repos) → 96.0% [bash-only scaffold] Claude Fable 5 (Jun 2026) → 95.0% [vendor-tuned agent] Claude Opus 4.8 (May 2026) → 88.6% [vendor-tuned agent] GPT-5.5 (Apr 2026) → 82.6% [vendor-tuned agent] Claude Opus 4.6 (Mar 2026) → 80.8% vendor-tuned / 75.6% bash-only Claude 4.5 Opus high reasoning → 76.8% [bash-only scaffold] Gemini 3 Flash high reasoning → 75.8% [bash-only scaffold] GPT-5.2 → 72.8% [bash-only scaffold] The last three rows are the apples-to-apples ones: same scaffold as mine, but on the full 500-task set across 12 repos. My row covers only a 50-task django+sphinx subset, so it's an easier set than every other row here — which is exactly why a free model sitting above Fable 5 deserves suspicion rather than applause. On HAL's leaderboard, which uses this exact mini dataset but a different scaffold, the top entry is ~72% (SWE-Agent + Claude Sonnet 4.5, older generation). —— WHY YOU SHOULD TAKE THIS WITH A GRAIN OF SALT (AND WHY I'M POSTING ANYWAY) —— Mini-50 only contains django and sphinx. These are two of the most heavily represented repos in LLM training data, so memorization/contamination could be inflating the score. One genuinely reassuring fact: per OpenAI's own human difficulty annotations, the mini-50 subset matches the full 500-set difficulty distribution (it actually holds slightly MORE >1-hour tasks, 16% vs 9%) — so this isn't "the easiest 50." n=50. That's roughly ±3pp sampling noise even before any contamination concerns. Every task is a 2019–2022 PR. Nothing remotely fresh; modern models have likely seen these issues in training. Free endpoint. I can't fully control or verify the serving stack, rate limiting, or whether responses get cached upstream. Baselines aren't apples-to-apples. As shown above, the 72–77% frontier numbers come from the full 500-task set across 12 repos, which is harder than django+sphinx alone. My 96% is on an easier subset — treat it as an upper bound. So the honest summary: ox-alpha-free is clearly VERY strong at agentic coding — first-tier behavior by every signal here — but treat 96% as an upper bound, not a comparable number. Happy to share configs, trajectories, and the scoring reports if anyone wants to dig in or reproduce. Also curious whether others have run any serious benchmarks on Ox Alpha — would love get more insights.