Gpt-6 Sol seems underwhelming..

As the title shows, GPT-6 Sol just seems kind of underwhelming to me. GPT-6 Sol is supposed to be the frontier coding agent and the numbers look pretty good on paper. GPT-6 Sol is much cheaper per input and output token than Astra. If you look at the actual benchmark charts, however, GPT-6 Sol Max seems to be roughly about the performance of GPT-6 Astra on medium on coding/agentic evaluations. However, if you look at the actual estimated cost per task rather than the price per token, the gap is surprisingly small. preview.redd.it/hf3sjqf154rh1.png preview.redd.it/0g6alwj354rh1.png Another thing to note, unless GPT-6 Sol has a much higher inference throughput (no benchmarks currently available), this would mean that tasks might take quite a bit longer to complete. So yeah, pretty underwhelming/disappointing to me initially

添加评论
点赞收藏
点踩分享查看原文
评论
?
参与讨论