Tensor Split Over PCIe 3.0 x1 — Good Results

When I was considering adding a second GPU, quite a few people told me PCIe 3.0 x1 would be a massive bottleneck and that upgrading my motherboard should be my first priority. I went ahead with the x1 riser anyway, and the results have been better than expected. Sharing as this likely have value to other planning machines. My home setup: Intel i5-14400F 2× RTX 3060 12GB (24GB total) One GPU on PCIe x16, the other on PCIe 3.0 x1 llama.cpp, running Gemma 4 31b Q5 (tight fit, tight context) Switching from layer splitting to tensor splitting increased +42% t/s (14 -> 20), even with one GPU connected over PCIe 3.0 x1. I did not do extensive testing yet though on this exact setup as I am waiting for a PSU upgrade before running those cards too hard. Based on what everyone said I expected it to crawl and was just planning on testing memory allocation... But then got the surprised positive result. Testing on I rented a machine with 2× RTX 5060 Ti 16GB (32GB total) to do some testing on some 70b models but decided to check PCIe traffic during tensor split before I shut it down. I was wondering how much bandwidth tensor splitting actually needed over PCIe. Test configuration: Model: L3.1 Nemotron Sunfall 71B IQ3_XS Context: 2k Flash Attention enabled llama.cpp PCIe Gen 3 capable links, running at x8 width Layer vs. tensor splitting: Test Layer Tensor prompt generation 11.7 t/s 20.7 t/s prompt processing 591.8 t/s 464.4 t/s +77% t/s with tensor splitting. PCIe bandwidth measurements I monitored PCIe traffic during tensor-split inference using: nvidia-smi dmon -s t -d 1 Here are some actual samples during generation (MB/s): GPU RX TX 0 92 102 1 96 105 0 94 108 1 107 108 0 92 100 1 94 99 0 98 107 1 105 111 0 93 103 1 95 101 0 92 99 1 94 98 Both GPUs consistently transferred approximately 90–110 MB/s in each direction during decode. PCIe 3.0 x1 has a theoretical bandwidth of approximately 985 MB/s per direction, this very likely not going to be a bottle neck issue for tensor split for low-mid range GPUs. Model loading would be slightly slower of course, on model loading I saw a spikes exceeding 4,000 MB/s. Obviously, x1 would slow that down adding several seconds to model load time. I am not switching models often though so a few extra seconds on load isn’t an issue. These results don't prove PCIe x1 has no performance penalty. The Vast cards weren't running over x1, and average throughput doesn't capture latency, synchronization overhead, or short transfer bursts. However, it looks very promising that upgrading my motherboard or sacrificing a nvme slot is probably unnecessary.