Have an Agent Babysit Your Deployments

Deployments are scary. That’s the moment you break things.

Not-deployments are even scarier. Waiting just makes the next deployment bigger.

As the saying goes: “If it hurts, do it more.”

The obvious, correct answer is CD. But then you’re off building canaries and waves and automated detection systems as gates. Canaries and waves are easy. Automated detection systems are hard. There are an indefinite number of things that can go wrong, and missing one of them takes you down. The asymmetry there is exactly the same asymmetry that makes deployments scary in the first place.

The historical answer was: It’s a lot of engineering effort and a lot of pain. And so CD gets delayed, and humans babysit deployments, and deployments happen infrequently, and the cycle of inefficient misery and fear continues.

This has exactly the right shape for an agent instead of code: Lots of rich data, a very long tail of possible states, relatively few runs (a handful a day, not 100qps).

And we now have intelligence on tap. Let’s use it!

At exe, Athena oversees our deployments. (All our bots have names, but that’s just so it’s easy to talk about them. They’re programs, not people.)

Athena sits in a system called “exe-ops” which is our Deployment Command Center. We started with shell scripts, but then built a UI. Traditionally, you do a migration to something like Spinnaker. Instead, we’re building up from shell scripts into the exact shape we want. Athena is part of that story.

The bot has read access to git and metrics and logs. It decides at each stage: should we proceed? Which machines should be in the next wave? It can escalate to a human and it can pause a deployment—or refuse to start one, if it deems it unwise. It communicates by sending us Slack messages.

It’s great! It is diligent and thorough. It reads the diffs, analyzes the logs, checks for unforeseen issues, self-heals around weird problems, and reports on how to make future runs smoother.

I could probably oversee deployments better than Athena. But the important question is not “in theory, could I do a better job?” but “in reality, will I do a better job?” We’re all busy. Athena does a much, much better job than I actually would.

Athena lets me focus my attention elsewhere, until something happens that’s worth my intervention. And by deploying more often, those interventions are rarer and smaller.