I removed 85% of gemma's weights (and it is mostly fine)

I pruned Gemma 4 26B-A4B, an MoE model, down to 15% of its original size, and the model still does great on one narrow task. It gets 92% agreement with the original on a classification task. That’s close to how often the 12B agrees with the 26B on the same ticks, which is 90%. This isn’t a model for general tasks, and tasks that require knowledge suffer because the knowledge is just gone. It was made for my app that runs an on-device LLM 24/7, so that makes memory footprint and energy consumption extremely important. On my Mac it hits 79 tokens/s, which is roughly double the speed of the 12B, using 3.8 GB of memory (roughly half). I selected a set of 8 experts per layer and used them for everything. Since the model selects 8 experts per token by default and there are only 8, the router no longer chooses which experts to use, and the model becomes dense: 3.81B parameters, 2.75 GB at Q4_K_M. What I want is for each block to do the same computation and produce the same result, given the same input. With block-local training, I’m not training on the answer. I’m training on the thought process. I’m saying, do the same thinking, not get the same answer. This is all trained off my own data. The method is efficient enough that it could be trained on a user’s own device, on their own data, in about 5 GB of memory, to mitigate issues stemming from deployment having different data than was in the training dataset. I haven’t tested that on a real user’s device yet. If you are curious about why I decided that fine tuning was a bad idea or how I designed this, take a look at the post I wrote going into the details of how I made this and why I felt this route was right for my application. This is the first time posting publicly like this. I would love feedback or thoughts, and to anyone who knows of better methods for choosing which experts to keep, please lmk. Post: tolby.app/blog/frankensteining-gemmas-experts