tokenizers v1 (rust)

localllama (Reddit)，r/LocalLLaMA，约 72.8 万成员的社区，专注于在本地硬件上运行大语言模型，涵盖模型选择、GPU 配置、量化技术、Ollama 与 llama.cpp 等工具，以及隐私优先的 AI 工作流。关注

Hey all! I am Aritra from Hugging Face. I wanted to share an update on the tokenizers library that we have at Hugging Face. It has gone under major changes and we have finally released version 1 of it. Here are what we are most excited about: > multiple language support > multi-thread scaling > minimal package size Read: huggingface.co/blog/tokenizers-v1

添加评论

点踩

分享查看原文

参与讨论