Training transformers where every layer W = V·Uᵀ from initialization reveals a corpus-determined optimal rank - looking for arXiv endorser (cs.LG) [D]

I've been running an experiment I'm calling Native Factorized Weights (NFW): instead of training a standard transformer and compressing it afterward, every linear layer is replaced by W = V·Uᵀ at initialization and trained that way from scratch. No post-hoc SVD. No LoRA adapter on top of a pretrained model. The factorization is the basis for the model and earns the name "Sliver layers" as each layer is comprised of two low-rank matrices. A comparison of a dense W matrix using n^2 parameters, vs. U and V com