Qwen4-27B's 4-way HC shift?
I just saw a PR in vllm-xpu-kernels today, revealing a significant architectural change in Qwen4-27B, and now that page shows 404 not found? It looks like a 4-way parallel residual stream to improve information flow and representation density within a fixed 27B parameter budget. (This means, instead of a single residual path, 4 parallel streams flow through the layers at the same time, maximizing performance per parameter. Best tech for 27b). QSA? maybe useless for small 27b. So anyway, I'm more expect 27b than flash MoE version. If effective depth is extended, I think we could see a massive leap in multi-step reasoning, coding, and reasoning consistency. ( see this if you want read more)
参与讨论