I need to keep some Slack connections alive. This should be easy.
Okay, but I have multiple replicas. And it gets slightly worse: the number of physical connections is not equal to the number of replicas, and it is not equal to the number of business-level connectors either. Several business connectors may share one external identity. One external identity may require several physical connections. And those connections need to be spread across whatever replicas are currently alive and have capacity. So who actually owns the connection? Then the questions start piling up. What if one replica dies? What if it doesn’t die, but loses access to the database? What if two replicas race for the same connection? What if the event that was supposed to wake the right worker never arrives? What if most workers are already full? A few questions and 2 hours of midnight walk in headphones and ChatGPT later, I had leases, runtime slots, reconciliation, failover, capacity limits, and a distributed ownership problem on my hands. This is the architecture I ended up with, and I have mixed feelings about this design. On one hand, I’m proud that I managed to account for so many different edge cases. On the other, the whole thing feels worryingly complex. So I’d really value input from people who have built similar systems. What did I miss or do you see anything that can break? And most importantly: what can be simplified without losing the guarantees? The full architecture description is a bit lengthy: AEON NEON - Connector Runtime - by Jarek J. Many thanks if you decide to read it and share your thoughts. I’d really like this one not to become another failed experiment.