quicktok: a faster tokenizer (exact and byte-identical with tiktoken) [P]

Been working on this a while! Should be useful for anyone trying to speed up their tokenization workflows. quicktok is a fast/exact BPE tokenizer written in C++. Token ids are byte-identical to tiktoken and encoding runs 2–3.6× faster than bpe-openai (the fastest alternative I know of) and 4–11× faster than tiktoken itself. It ships cl100k, o200k, GPT-OSS, Llama-3, and Qwen2.5/3. Approach. Same algorithm as bpe-openai (exact backtracking BPE) but I apply lots of data structure engineering to cut memory acce