How are people hosting random GGUF / open models behind an API?
I keep running into this annoying gap: A model exists on Hugging Face. Sometimes it has GGUFs. It runs fine locally in Ollama / llama.cpp / LM Studio. But if I want to use it from an actual app, there is no hosted API for it. The usual answers seem to be: - run it locally, which is fine for personal use but not really an app backend - rent a GPU and serve it myself - deploy vLLM / llama.cpp server on Modal, RunPod, Baseten, Replicate, etc. - hope OpenRouter / Together / Fireworks / DeepInfra already has tha
评论
?
参与讨论