我用元胞自动机造了一个语言模型
作者用元胞自动机（每个格子持一个向量，反复和 8 个邻居交互，无注意力、无 Transformer 块）搭了一个语言模型，想验证仅靠局部邻域通信能否涌现出下一个字符预测能力。 模型写出的第一句看起来像句子的内容是 “the little red fox ran home.”，但随后就退化成重复片段，说明纯局部交互很难稳定生成连贯文本。 这是一篇偏实验手记的帖子，信息量集中在“尝试和结果”，适合对替代架构、可解释性或自组织系统感兴趣的读者。
评论
点赞
收藏LLM 推理背后的记忆、调度与流式传输系统
<p>当我刚开始研究时，我以为推理过程基本上就是这样的：<br>while True: logits = model(tokens) token = logits[-1].argmax() tokens.append(token)，从技术上讲，这没错。<br>然而，这并不能说明当两个人同时使用模型时会发生什么，也无法解释模型如何保存所有历史 token，更无法解释为什么 ChatGPT 在完整回复尚未生成时就开始输出了。<br>Transformer 只负责预测下一个 token。在其外围还有一整套系统，负责记忆历史状态、决定接下来该为谁生成哪个 token，并将该 token 发送给对应的用户。</p>
MoE路由本质上就是分支预测
<p>如果你同时涉足系统编程和机器学习基础设施领域，读到关于 Transformer 推理的内容时，就会产生一种奇妙的似曾相识之感。<br>术语是新的：KV 缓存、专家路由、分页注意力机制。但问题的本质却并无二致。<br>三十年前，CPU 架构师就遇到了同样的瓶颈：你必须在弄清需求之前就采取行动。<br>本文将探讨 Transformer 推理流程中两个几乎原封不动地重现这一难题的环节。</p>
为什么 malloc 分配的内存总是比我要求的更多？
作者从零实现C语言内存分配器，拆解malloc背后的Header、Back Pointer、Padding对齐及内部碎片机制。通过First-fit、Splitting和Coalescing算法解决空闲链表管理问题，并分享调试中对齐错误与内存损坏的踩坑经历，清晰展示底层内存管理的权衡与复杂度。
NPM sucks for a solo dev
i spend every week doing something that has absolutely nothing to do with putting out good content for my blog. i am fixing a transitive dependency vulnerability in a package i didn’t even know i was ...
I spawn random interests and I don't know why
Sometimes I wish that I had something to stick onto like a niche or a skill that I can keep doing for the rest of my life. But it had never come. I still keep stumbling upon things that are tangential...
FPGAs broke my brain
an FPGA engineer recently used a tamil movie analogy to explain hardware design, and it fundamentally changed how i approach programming. he compared FPGAs to Anniyan-the chaotic protagonist with thre...
winning without a battle
i was literally screaming my lungs out - “AKAZAAAAA” - along with the protagonist when akaza showed up before tanjiro in the Demon Slayer: Infinity Castle but something about their battle lingered. ap...
Welcome to My Blog
Welcome! This is the first post on my new blog, and I’m excited to share my thoughts with you. What to Expect This blog will cover a variety of topics that interest me, including: Technology & Enginee...