I took EmbeddingGemma-2 (577M) from Rank #26 to #1 on the MTEB Audio leaderboard (MAEB), beating 9B models by adding a 2.4 MB PCA head with zero training

Model & weights: BrunoHays/embeddinggemma-2-nullspace-audio MAEB (Massive Audio Embedding Benchmark) is the main MTEB leaderboard benchmark for audio and text-to-audio embeddings, currently dominated by 8B-9B multimodal models. I took Google's edge-sized google/embeddinggemma-2 (577M params, 768d vectors) and boosted its MAEB score from 49.3% to 54.4% (+5.1 points) , jumping from Rank #26 to #1 on MAEB audio-only (and #16 to #1 on full MAEB ) without training a single weight. The whole addition is a 2.37 MB safetensors file of frozen PCA matrices. Why this matters for local / edge setups At 577M parameters and 768d vectors (15x smaller than 9B Omni models), it runs easily on a laptop or edge device. Standard text-aligned audio models are great at matching spoken words to text, but terrible at grouping audio by voice, emotion, or sound similarity because the final layers keep what words were said and drop how the sound was made . By pulling features from earlier layers of the frozen audio encoder, this model keeps the base model's 100+ language text-to-audio search while fixing audio-to-audio tasks: Unsupervised voice/gender clustering: 0.05% -> 63.1% Artist / song audio-to-audio search: 55.2% -> 93.8% Vocal emotion classification: 19.7% -> 40.2% How it works Early layers still know acoustics: Speech representation work ( Pasad et al. ; Bouziane, Mdhaffar & Estève ) shows that early/middle layers keep speaker, pitch, and emotion traits that disappear in the final layers. In EmbeddingGemma-2, Layer 2 and Layer 6 still hold those signals. 256 empty directions in the 768d vector: EmbeddingGemma-2's text backbone is actually 512d, projected up to 768d at the end by a 768 x 512 linear matrix. That leaves 256 perpendicular directions (the null space) that text vectors never touch. Filling the null space with PCA: I compressed Layer 2 and Layer 6 down to 256 dimensions with PCA (fitted on 1,400 external audio clips) and injected them into those 256 unused directions. Caveats: Because there is no retraining, audio embeddings need all 768 dimensions (no Matryoshka truncation to 512/256/128), and adding unweighted acoustic features hurts some purely spoken-intent tasks (e.g. MInDS14 drops 97.6% -> 88.3% ). It shows how much untapped signal sits inside today's overtrained models, even with zero training. Full breakdown and usage in the Hugging Face model card .