GitOps¶

GitOps is the practice of using Git as the single source of truth for declarative infrastructure. We use Argo CD to continuously reconcile the cluster to match the desired state in our infrastructure repository. If the current state of the cluster drifts from the desired state declared in Git, Argo CD automatically synchronizes the cluster. This also means that every change to the cluster is a commit, which gives us auditable history and easier rollbacks.

Argo CD¶

We run the high availability install of Argo CD and bootstrap it with a single ApplicationSet. It scans the project directories in our repository and creates an Argo CD application for each directory it finds, with automatic synchronization enabled. To deploy something new to the cluster, we simply commit a new directory.

Here is an example of a simple application deployed with Argo CD:

Manifests¶

All of our manifests are composed with Kustomize. Any upstream Helm charts are rendered inline with Kustomize, so chart versions and values are stored in Git along with the rest of our resources. We also built a custom Kustomize generator that is a containerized KRM function which renders Jsonnet configuration. We use it with kube-prometheus to generate our monitoring stack¹.

Secrets¶

We encrypt secrets with Sealed Secrets so that they can be safely committed to a public repository. A SealedSecret can only be decrypted by the controller running in our cluster. For secrets that are built from values already present in the cluster, we use External Secrets to compose them declaratively. We also manage some resources outside of the cluster with Crossplane, such as the Microsoft Entra ID app registrations for SSO with OAuth2/OIDC.

Automated Image Updates¶

Argo CD Image Updater watches our container registries for new container images and commits tag/digest updates back to the repository. This way, the repository remains the source of truth even for automated changes.