Skip to content

Zero Kubernetes Secrets Architecture

CloudTaser eliminates Kubernetes Secrets from the trust chain -- not only for customer workloads, but for its own operational secrets. The operator, webhook TLS certificates, broker tokens, and enrollment credentials all live in vault and are loaded directly into memory at runtime.

Why

CloudTaser's core promise is that secrets never touch etcd. Storing its own operational secrets in Kubernetes Secrets would undermine that promise in several ways:

  • Dogfooding. If CloudTaser protects customer secrets by keeping them out of K8s Secrets, it should do the same for its own. An operator that stores its webhook TLS key in a Kubernetes Secret is one kubectl get secret -o yaml away from compromise.
  • etcd as an attack vector. etcd backups, etcd snapshots, and etcd encryption-at-rest key management are all surfaces where operational secrets could leak. Removing them from etcd eliminates the surface entirely.
  • Unified audit trail. When all secrets (customer and operational) go through vault, every access is recorded in the vault audit log. There is no second audit path to maintain.
  • Sovereignty for operational secrets. CloudTaser's own TLS certificates and broker tokens deserve the same EU-sovereign storage as customer data. If the vault is in the EU, so are the operator's secrets.

How It Works

Authentication

The operator authenticates to vault using the Kubernetes auth method. It presents its ServiceAccount JWT token to vault, which validates it against the cluster's TokenReview API. No Kubernetes Secret is involved -- the ServiceAccount token is projected into the pod by the kubelet automatically.

Operator Pod
  |
  |  1. Read projected ServiceAccount JWT from /var/run/secrets/kubernetes.io/serviceaccount/token
  |
  v
Vault (K8s auth method)
  |
  |  2. Validate JWT via TokenReview API against the cluster
  |
  v
Return vault token (short-lived, scoped to cloudtaser-operator policy)

First Install

On first startup, when no secrets exist in vault yet, the operator bootstraps itself:

  1. Generates a self-signed CA and webhook TLS certificate
  2. Generates broker mTLS certificates (server, client, CA)
  3. Generates a random broker authentication token
  4. Writes all of these to vault at secret/cloudtaser/system/*
  5. Loads them into memory and begins serving

No Kubernetes Secret is created at any point.

Subsequent Starts

On subsequent startups, the operator:

  1. Authenticates to vault via Kubernetes auth
  2. Reads existing secrets from secret/cloudtaser/system/*
  3. Loads them into memory
  4. Begins serving the webhook and broker with the loaded credentials

Webhook TLS

The webhook TLS certificate is served from memory using Go's tls.Config.GetCertificate callback. There is no file mount and no Kubernetes Secret containing the certificate. The API server's MutatingWebhookConfiguration is patched with the CA bundle on every operator startup.

tlsConfig := &tls.Config{
    GetCertificate: func(*tls.ClientHelloInfo) (*tls.Certificate, error) {
        return operatorCertManager.GetCurrent(), nil
    },
}

Certificate Rotation

The operator checks certificate expiry periodically (default: every hour). When a certificate is within 30 days of expiry:

  1. Generates a new certificate signed by the existing CA
  2. Writes the new certificate to vault
  3. Updates the in-memory certificate (the GetCertificate callback returns the new one immediately)
  4. Patches the MutatingWebhookConfiguration if the CA changed

No restart required. No Kubernetes Secret rotation.

High Availability

When the operator runs with multiple replicas (operator.ha: true), all replicas need access to the same broker token so they can authenticate wrapper connections consistently. Vault provides this naturally:

  • All replicas read the shared broker token from secret/cloudtaser/system/broker-token
  • Leader election determines which replica actively serves the webhook
  • The standby replicas have the same credentials loaded and can take over instantly

This replaces the pattern of storing shared state in a Kubernetes Secret and watching for changes.

Vault Paths

All operator secrets are stored under a single vault path prefix:

Path Contents
secret/cloudtaser/system/webhook-tls Webhook server certificate, private key, and CA certificate
secret/cloudtaser/system/broker-tls Broker mTLS certificates: server cert + key, client cert + key, CA certificate
secret/cloudtaser/system/broker-auth Broker authentication token (used by wrappers to authenticate to the broker)
secret/cloudtaser/system/broker-token Shared broker token for HA replicas (ensures consistent wrapper auth across failover)
secret/cloudtaser/system/enrollment SaaS platform enrollment token (only present when connected to CloudTaser Platform)

All paths use the KV v2 secrets engine. The operator's vault policy grants CRUD access to secret/data/cloudtaser/system/* and secret/metadata/cloudtaser/system/*.

The One Exception

The vault bootstrap credentials -- unseal keys and root token -- cannot be stored in the vault they unlock. This is an inherent constraint of any vault-based system.

During cloudtaser-cli source install + cloudtaser-cli source configure, these credentials are:

  1. Output once to the terminal
  2. Must be stored externally in an HSM, on paper (Shamir shares), or in an external KMS
  3. Deleted from Kubernetes after retrieval (if they were temporarily stored during automated bootstrap)

Unseal keys are the one secret CloudTaser cannot protect

The vault unseal keys and initial root token are the single exception to the zero-K8s-secrets model. Treat them with the highest level of care. Do not store them in the same cloud account as the vault. Do not store them in a Kubernetes Secret. The source install command outputs them exactly once.

Fallback Mode

For environments where vault is not available at operator startup, the operator supports falling back to Kubernetes Secrets:

operator:
  secretBackend: kubernetes  # default: "vault"

Or via environment variable:

CLOUDTASER_SECRET_BACKEND=kubernetes

When secretBackend is set to kubernetes, the operator uses the existing behavior: generating certificates and storing them in a Kubernetes Secret (cloudtaser-operator-certs).

When to use fallback mode:

  • Local development with kind, minikube, or k3d where no vault is available
  • Environments where vault is not reachable at operator startup (vault deployed after the operator)
  • Gradual migration from older installations that already have certificates in Kubernetes Secrets

Fallback mode weakens the security model

With secretBackend: kubernetes, operational secrets are stored in etcd. This means they are subject to the same risks CloudTaser protects customer secrets against: etcd backup exposure, cloud provider access, and lack of centralized audit. Use vault mode in production.

Setup

The cloudtaser-cli source configure command configures vault for operator secret storage:

cloudtaser source configure \
  --openbao-addr https://vault.eu.example.com \
  --token hvs.YOUR_ADMIN_TOKEN

This creates:

  1. Vault policy cloudtaser-operator -- grants CRUD on secret/data/cloudtaser/system/* and secret/metadata/cloudtaser/system/*
  2. Kubernetes auth role cloudtaser-operator -- bound to the operator's ServiceAccount (cloudtaser-operator in namespace cloudtaser-system)

After running this command, deploy or restart the operator with secretBackend: vault (the default). The operator will authenticate, bootstrap its secrets on first run, and serve from memory on subsequent runs.

Preview first

Add --dry-run to see what will be created without making changes: 

cloudtaser source configure --dry-run \
  --openbao-addr https://vault.eu.example.com \
  --token hvs.YOUR_ADMIN_TOKEN

Security Model

The following diagram shows the trust chain with vault-based secret storage. The key property is that no Kubernetes Secret appears anywhere in the chain.

flowchart TD
    subgraph "Kubernetes Cluster"
        SA["ServiceAccount Token<br/>(projected by kubelet)"]
        OP["CloudTaser Operator"]
        WH["Webhook Server<br/>(TLS from memory)"]
        BR["Auth Broker<br/>(mTLS from memory)"]
        API["K8s API Server"]
    end

    subgraph "EU Vault"
        KA["K8s Auth Method"]
        SS["secret/cloudtaser/system/*"]
        AL["Audit Log"]
    end

    SA -->|"1. JWT token"| OP
    OP -->|"2. Login with JWT"| KA
    KA -->|"3. Validate via TokenReview"| API
    KA -->|"4. Return vault token"| OP
    OP -->|"5. Read/write secrets"| SS
    SS -->|"6. Load into memory"| OP
    OP --> WH
    OP --> BR
    SS -.->|"All access logged"| AL

    style SA fill:#e8f5e9,stroke:#2e7d32
    style SS fill:#e3f2fd,stroke:#1565c0
    style AL fill:#fff3e0,stroke:#e65100

What this eliminates:

Traditional approach Zero K8s Secrets approach
Webhook cert stored in cloudtaser-operator-certs K8s Secret Webhook cert stored in vault, served from memory
Broker token stored in a K8s Secret Broker token stored in vault, loaded into memory
etcd contains CloudTaser TLS private keys etcd contains no CloudTaser secrets
kubectl get secret exposes operator credentials kubectl get secret returns nothing CloudTaser-related
Cert rotation requires Secret update + rollout Cert rotation is in-memory, no restart needed
Audit requires K8s audit policy configuration All access logged in vault audit log automatically

Trust boundaries:

  • The operator trusts the kubelet to project a valid ServiceAccount token
  • The operator trusts vault to authenticate and authorize correctly
  • Vault trusts the Kubernetes TokenReview API to validate ServiceAccount tokens
  • No trust is placed in etcd, Kubernetes Secrets, or the cloud provider's control plane