How do you catch a quality drop on your own task when the model updates under you?

I run a small product where the model does the actual work on whatever the user uploads, and I have no labeled eval set. Every time the version underneath me changes I find out from a user before I find out from testing, usually because some input I never try comes back in a shape the rest of my pipeline was not written for. Hand reading outputs once a week is the thing I always intend to do and never keep up with. The fix I keep coming back to is a fixed folder of inputs that cover the cases I actually care about, run it after every update and compare against the last set of outputs I trusted. The problem is that the comparison tells me something changed, not whether it got worse. On long text outputs the diff is mostly noise and I end up reading it anyway, which lands me back at judging by feel. So I am curious how people at this size handle it. Do you keep a golden set and score it with rules, do you have a second model grade the outputs, or do you freeze a version and only move when something forces you? And when the drop lands on a task you cannot fully specify, is fixing the prompt usually enough or is the honest answer that the smaller model is done?