What is the hardest part of manufacturing your first product?(I will not Promote)

I've been talking to a few people who are building physical products, and I'm noticing that actually designing the product isn't always the hardest part. things like finding manufacturers who work with smaller quantities , Qualtiy control , reasonable MOQ etc can be some major problems For those of you who've actually taken a physical product into production: What was the single biggest manufacturing headache you ran into? I'm particularly interested in experiences from people doing their first 100–5,000 units.