Width vs. Depth: Speculating on the Margin

Some thinking about how to trade off batching and speculative decoding in a running inference engine.