I built a native Vulkan training backend for 143 modern Transformer architectures — no CUDA or PyTorch required

I've been working on something that started as the training backend for my Hierarchos architecture, but it has grown into a much broader project: github.com/necat101/Hierarchos-Native Hierarchos Native now includes a native Rust + Vulkan backend for training and inference across 143 canonical Transformer architectures , with additional Hugging Face model_type aliases bringing the currently advertised registry to 224 model-type spellings . The part I'm most interested in sharing isn't actually Hierarchos itself anymore. It's the Transformer backend underneath it. The goal is to explore how much of the modern Transformer ecosystem can be implemented directly on Vulkan compute rather than assuming CUDA + PyTorch as the default execution stack. The backend currently includes native implementations for things such as: forward and backward passes full-model training AdamW LoRA/PEFT-style fine-tuning embeddings and LM heads attention and KV-cache paths dense and MoE architectures RMSNorm / LayerNorm RoPE and positional operations common activation functions cross-entropy and gradient computation mixed-precision infrastructure checkpointing/resume SafeTensors model interchange Hugging Face model, tokenizer, and dataset acquisition from Rust native generation/inference paths The supported architecture registry covers families including Llama, Qwen, Gemma, Mistral/Mixtral, DeepSeek, Phi, OLMo, Granite, Cohere, BERT-family models, T5/BART-family models, and many others. There are also architecture-specific Vulkan kernels for newer model designs rather than attempting to force everything through one generic approximation. One thing I want to be careful about: I'm not claiming this is currently a CUDA/PyTorch replacement, or that supporting a model type automatically means every possible configuration and multimodal component of that model has been validated. The architecture registry describes native text-graph implementations. Multimodal packages may contain vision/audio/etc. towers that aren't part of the native text backend, and real-world compatibility still needs broader testing across models and hardware. What I do think is interesting is the direction this demonstrates. Vulkan is available across NVIDIA, AMD, Intel, and a lot of integrated/mobile-class hardware. If native training stacks like this can mature enough, there is potentially room for ML training infrastructure that is substantially less tied to a single vendor compute ecosystem. I've been developing and testing this on AMD RDNA 3 hardware, including a ROG Ally Z1 Extreme, which has also been useful for exposing assumptions that tend to go unnoticed when ML software is developed exclusively around high-end CUDA systems. The repository includes the Rust source, Vulkan compute shaders/SPIR-V, CLI, Hierarchos runtime, architecture documentation, validation utilities, and standalone packaging support. The supported architecture list is here: hierarchos-vulkan/README_ARCHITECTURES.md And the main README now contains examples for building the backend, checking Vulkan devices, training Hugging Face models, LoRA fine-tuning, inference, and training Hierarchos itself. I'm especially interested in feedback from people working on: Vulkan compute, ML compiler/runtime design, GPU kernels, PyTorch alternatives, AMD/Intel ML, portable training systems, or Transformer architecture implementations. There is still a lot that needs benchmarking, hardware validation, optimization, and independent testing before making strong performance claims. But getting this many modern architecture graphs running through a native Vulkan-oriented training system has made me think there may be something genuinely useful here beyond the original Hierarchos project. Repo: github.com/necat101/Hierarchos-Native I'd be very interested in technical criticism, architecture compatibility reports, benchmarks on other GPUs, or contributors who want to stress-test the backend.