The cut in the Mixture of Experts compute graph
The MoE backward pass hides a small trick that, if you miss it, means you'll never train. Let's derive it by hand to see why.
点赞
收藏
点踩
分享评论
?
参与讨论
The MoE backward pass hides a small trick that, if you miss it, means you'll never train. Let's derive it by hand to see why.