Flux and OpenBao: Secrets and Signatures
GitOps helps us declare our desired workloads, but how do we deal with and manage secrets? Additionally, as our fleet grows, we also blend artifacts and configuration from many different sources. How do we trust what we are running?
OpenBao is an open source secrets and encryption platform under the OpenSSF. In this post we'll integrate OpenBao with Flux in two ways:
- kustomize-controller will decrypt SOPS-encrypted Secrets through OpenBao using workload identity, with no static
BAO_TOKENorVAULT_TOKENto bootstrap - Cosign will sign OCI artifacts with a key held within OpenBao, producing signatures Flux can verify without any service outside your infrastructure
For both integrations, we'll use two OpenBao features. The Transit secrets engine performs encrypt, decrypt, and sign operations without ever releasing the key material, acting as a Key Management System (KMS), and the Kubernetes and JWT auth methods let a workload trade its Kubernetes-issued ServiceAccount token for a short-lived OpenBao token, so no long-lived credential has to exist on either the OpenBao or Kubernetes side.
Configuring SOPS: GitOps secrets without a static token
Keeping encrypted secrets in Git solves most of the secret-distribution problem: the ciphertext goes through the same pull requests and the same reconciliation as the rest of your configuration. SOPS, a CNCF project, is the standard tool for this. It encrypts the values of a file with a data key, then asks a KMS to protect that data key; only the small data key ever travels to the KMS. OpenBao is one of many supported backends that SOPS can use, and many teams rely on Flux's support for decrypting secrets with SOPS.
Bootstrapping secrets management is tricky though. The SOPS encryption key lives in OpenBao's Transit engine, so Flux used to need a static BAO_TOKEN added in the cluster in order to request decryption. Teams installing Flux would need to provision this bootstrap secret before using GitOps to manage secrets. Starting with Flux v2.9, that token is optional: kustomize-controller can now directly authenticate to OpenBao with Kubernetes ServiceAccount tokens.
Step 1: Configure OpenBao for SOPS decryption
For our use-case, OpenBao needs a Transit key, a decrypt-only policy, and an auth role bound to the kube ServiceAccount Flux will use. Both the "Kubernetes" and "JWT" auth methods within OpenBao work for this; with Kubernetes auth and the flux-system/kustomize-controller ServiceAccount, the setup is:
bao secrets enable transit
bao write transit/keys/sops key_type=aes256-gcm96
bao policy write flux_sops_decrypt - <<EOF
path "transit/decrypt/sops" {
capabilities = ["update"]
}
EOF
bao auth enable kubernetes
bao write auth/kubernetes/config \
kubernetes_host=https://kubernetes.example.com:6443 \
kubernetes_ca_cert=@ca.crt \
token_reviewer_jwt=@reviewer.jwt
bao write auth/kubernetes/role/flux-system_kustomize-controller \
bound_service_account_names=kustomize-controller \
bound_service_account_namespaces=flux-system \
audience=https://openbao.example.com:8200 \
token_policies=flux_sops_decrypt \
ttl=20m
The role name here follows Flux's {namespace}_{name} convention, and the audience needs to match the OpenBao address that will show up in the SOPS metadata below. Note that flux_sops_decrypt only grants decryption; the developer or CI identity encrypting in Step 2 needs update on transit/encrypt/sops instead. The Kubernetes auth and JWT/OIDC auth API docs cover the OpenBao server-side options.
Step 2: Encrypt a Secret with OpenBao
After authenticating to OpenBao, a developer or CI job encrypts a regular Kubernetes Secret with the Transit key:
sops encrypt \
--hc-vault-transit https://openbao.example.com:8200/v1/transit/keys/sops \
--encrypted-regex '^(data|stringData)