The Best Way to Use Text Embeddings Portably is With Parquet and Polars

Text embeddings , particularly modern embeddings generated from large language models, are one of the most useful applications coming from the generative AI boom. Embeddings are a list of numbers which represent an object: in the case of text embeddings, they can represent words, sentences, and full paragraphs and documents, and they do so with a surprising amount of distinctiveness. Recently, I created text embeddings representing every distinct Magic: the Gathering card released as of the February 2025 Aetherdrift expansion: 32,254 in total. With these embeddings, I can find the mathematical similarity between cards through the encoded representation of their card design, including all mechanical attributes such as the card name, card cost, card text, and even card rarity. The iconic Magic card Wrath of God , along with its top four most similar cards identified using their respective embeddings. The similar cards are valid matches, with similar card text and card types. Additionally, I can create a fun 2D UMAP projection of all those cards, which also identifies interesting patterns: The UMAP dimensionality reduction process also implicitly clusters the Magic cards to logical clusters, such as by card color(s) and card type. I generated these Magic card embeddings for something special besides a pretty data visualization, but if you are curious how I generated them, they were made using the new-but-underrated gte-modernbert-base embedding model and the process is detailed in this GitHub repository . The embeddings themselves (including the coordinate values to reproduce the 2D UMAP visualization) are available as a Hugging Face dataset . Most tutorials involving embedding generation omit the obvious question: what do you do with the text embeddings after you generate them? The common solution is to use a vector database , such as faiss or qdrant , or even a cloud-hosted service such as Pinecone . But those aren’t easy to use: faiss has confusing configuration…