Applying Sliding Window Attention to pretrained LLMs at inference time [P]

I've been working on a practical implementation of Sliding Window Attention (SWA) for pretrained Hugging Face causal LLMs. The idea is simple: instead of allowing every generated token to attend to the complete historical KV cache, maintain a bounded cache consisting of: attention sinks + recent sliding window I implemented this as a reusable inference layer rather than modifying or retraining the model. GitHub: github.com/oraby8/SWA The implementation currently includes: bounded KV cache circular/ring-buffer storage attention sinks streaming prefill chunked attention masking autoregressive decoding Full Attention vs SWA benchmarking TTFT / TPOT / throughput measurements KV-cache memory measurements One interesting result from my Qwen2.5-7B experiment: Context Full KV SWA-64 16K ~923 MB ~3.5 MB 32K ~1.84 GB ~3.5 MB 64K OOM ~3.5 MB At 16K, SWA-64 also reduced TPOT from ~38.4 ms to ~30.5 ms in this setup. However, there is an important trade-off: tasks requiring information far outside the active window can degrade. I'm currently investigating how much of this is inherent to SWA versus implementation/model-specific behavior. I'm sharing the implementation mainly to get feedback from people working on LLM inference, KV-cache optimization, and long-context models . I'd be particularly interested in: Which model architectures should I validate next? What failure cases should I benchmark? What would make this useful for existing HF inference workflows? Are there cache/attention implementation details I may be overlooking? Feedback and experiments are very welcome.