Renting a moving truck to grab a pizza? Feedback on my OSINT pipeline architecture
Working on a personal pipeline that pulls public signals from a few sources (earthquakes, internet infrastructure status, disaster alerts, markets, etc.), compares each one against its own historical baseline, and runs on AWS with cache invalidation on every update. The goal from the start was for it to run on its own at the lowest possible cost, without buying hardware I'd have to babysit like a pet. (I already have a cat for that.) No boxes to maintain, no drives to worry about. Question for people who've done this longer than me: does this sound reasonable, or is it over-dimensioned? It feels a bit like renting a full moving truck just to go grab a pizza — a lot of infrastructure (automated deploys, CDN invalidation, EventBridge, Lambda, Glue, State Machine) for something that's fundamentally just pulling public APIs and comparing against a historical window. Would something much simpler (cron job + script + a static JSON file) get the same result, or does the fuller stack genuinely pay off once you're running several signals continuously?