Generative Retrieval: Tokenizers and Semantic IDs

A practical guide to residual quantization, RQ-KMeans, RQ-VAE, codebook balancing, and the design and evaluation of semantic IDs for generative retrieval.