Benchmark notes: V4.1 Flash 1017 improves both score and efficiency

I evaluated DeepSeek V4.1 Flash 1017 through the native DeepSeek API, at max reasoning effort, on MindTrial 's 98 tasks: 39 text and 59 visual. Python with scientific libraries is available, with a ten-call limit per task and strict answer validation. The result was 87 passes, seven failures and four errors . The more interesting change is the reduction in work needed to get there. Metric Previous V4.1 Flash 1017 Change Passes 81/98 87/98 +6 Hard errors 13 4 −9 Summed request time 4:17:27 2:35:50 −39.5% Output tokens, including reasoning 3,643,063 1,591,068 −56.3% Reported reasoning tokens 3,295,178 1,405,336 −57.4% Python executions 499 450 −9.8% Successful tool attempts 83.6% 90.2% +6.7 pp The recorded model settings are unchanged. The new run has 451 tool attempts but only 450 executions: one malformed-arguments attempt never starts Python. Most of the score improvement comes from recovering previous errors. There are nine newly passed tasks and three lost passes; eight of those gains were previously errors. The text score stays at 38/39, while visual passes rise from 43 to 49/59. It passes all nine OCR and all eight visual deduction-pattern tasks. Tool behavior improves too: nonzero exits fall 32→19 , and timeouts 45→24 . Including Python process time, the recorded time reduction is 43.6% . Even on the 78 tasks passed by both versions, request time falls 42.7% . One remaining error is clearly about the response contract: the model returns exactly 15 , matching the expected answer, but the parser requires a structured result object. The other three errors stop at length after ten Python executions. I preserved all four official errors. For context, Kimi K3.1 Pro scores 90/98 but takes 5:04:54 of request time. GPT-6.1 Sol scores 89/98 in 1:06:14. DeepSeek improves substantially without leading this suite on either score or speed. Request time includes inference/network time but excludes Python and validation.

参与讨论