Overview¶
Our hyperconverged, highly available Kubernetes cluster runs on 3 bare-metal machines with Talos Linux. All 3 nodes are a part of the Kubernetes control plane, the etcd cluster, and the Ceph cluster, and also allow scheduling as this is a fairly small deployment.
Key characteristics¶
- Hyperconverged: compute and storage share the same 3 nodes.
- Immutable OS: Talos Linux is API-driven and declarative, eliminating configuration drift.
- Highly available control plane: 3 control plane nodes + 3-member etcd cluster.
- Distributed storage: Ceph spans all 3 nodes to provide block, file, and object storage.
- GitOps-managed: cluster state is managed declaratively from git and continuously reconciled.
Operating System¶
Talos Linux is a secure, immutable, and minimal operating system for Kubernetes that removes configuration drift with infrastructure as code. All configuration for the OS is declarative and submitted to Talos' API. There is no SSH, no package manager, no shell; it is extremely minimal. This makes it the perfect OS to purely run Kubernetes.
Core components¶
- Networking (CNI): Cilium
- Networking (Service Mesh): Istio
- Certificates: cert-manager
- Storage: Rook Ceph
- Secrets Management: Sealed Secrets
- Observability: Grafana, Hubble, Kiali, Prometheus, Loki, Alloy, and more
- Infrastructure as Code: Crossplane
- GitOps: Argo CD
- Policy: Kyverno
- Databases: CloudNativePG (PostgreSQL), Redis Operator