Instead of pre-indexing a catalogue, I made the cache miss populate the database — architecture writeup

Building a cross-topic collection app (films, series, anime, books, games, restaurants) I hit a sourcing wall: there's no catalogue that spans those six with equal quality, and stitching six vertical providers together gives you six integrations, six failure modes, and still no row for the item none of them carry. So I treated the database as a cache of demand rather than a snapshot of a corpus. Read path: Query hits Postgres. Hit → plain DB read, no model in the path. This is theoverwhelming majority of requests once a topic has history. Miss → Supabase Edge Function (Deno) runs: Tavily search + fetch → Claude APIextracts into the target topic's schema → schema validation → insert with aconfidence score. The row is permanent and shared. Cost per item is paid exactly once. Two design calls worth discussing: Per-topic schemas over a generic item table . A film is director/runtime/year; a restaurant is cuisine/location/price band. One flattened shape makes everything unsearchable in the way its topic actually wants. It also gives the extraction step a real target instead of a bag of fields — which is precisely why a generic scraper wasn't enough. No human in the write path . Review-before-publish defeats the premise (the user wants this item now), so writes are optimistic: land immediately with a confidence score, users flag bad rows into an admin queue. Confidence scoring + flagging replaces gatekeeping. Known weaknesses, since they're the interesting part: Cache miss is seconds of wall clock. Unacceptable for a user who doesn't know what's happening behind the spinner. No bulk import of obviously-popular items and no batch expansion on a hit (found a film → pull that director's filmography while you're there). So a common query and an obscure one cost the same today. Usage limits contain that instead of fixing it. Stack: React 19 + TypeScript + Vite, Tailwind + shadcn/ui, Zustand, Supabase (Postgres/Auth/Edge Functions, pg_cron + pg_net), Cloudflare Pages. Source (MIT): github.com/Zartof23/mytops Live: mytops.io Would like to hear from anyone who's run demand-driven ingestion in production — particularly how you handled duplicate/near-duplicate entries at volume.