Thinking Machines dropped RoPE, and it's a good idea
Thinking Machines' first model encodes position with a learnt attention bias rather than RoPE, but it's not just a replacement, it's a fundamental shift in how position is thought of.
点赞
收藏
点踩
分享评论
?
参与讨论
Thinking Machines' first model encodes position with a learnt attention bias rather than RoPE, but it's not just a replacement, it's a fundamental shift in how position is thought of.