service ingress/egress control - easier options?
How have other people handled ingress/egress control for services? My setup: I have a flat network and a linux mini pc that primarily runs docker compose stacks. I expose a small number of services publicly - yes I have and use vpn tunnels, no I don’t want to use a vpn tunnel for these. I have hardened my docker services but network ingress/egress control has proven challenging. Docker network options: As I understand, docker has an internal network mode or a bridge network mode. Internal network mode is completely locked down and bridge network mode is wide open via the host interface and there is nothing in between. The problem: Many services I host require at least some egress - for example to reach an idp for odic login or a notification service - so an internal network is not feasible, nor is putting them in the same docker network as they need to connect via https. Otherwise many stacks have no reason to access the internet. My current solution: The solution I use is to create a custom arch based sidecar container in each stack to act as a stack netns. It uses a (vibe coded) bash script to resolve and set hostnames/iptable rules at startup fed in as an environment variable to act as gatekeeper for the stack. I set the rules for each stack in the compose file when setting up the stack. This is setup as an ‘internal+’ network - default deny with a whitelist, or an ‘external-’ network - default accept with a blacklist. Most stacks then get an effectively internal network or wan access but blocked from the lan. The net effect is no services can touch the lan. This works fine except when I forget to grant access to the idp and can’t work out why login fails or forget to point my proxy to the sidecar container, but comes with the downside of a custom setup, increased complexity, increased troubleshooting difficulty and a risk of port clashes as all stack services now share the same netns. My query: Does docker have some native way of solving this problem that I don’t know about, or am I expecting too much or approaching this the wrong way? I briefly looked into k3s as it seemed to have more features in this space but I run a single server and have no k8s experience. I primarily use rootless docker so macvlan’s/ipvlan’s and network rules on a switch or router are not an option. I have no idea about podman but believe it has the same limitations and I don’t really want to change entire setups for this relatively small issue alone.