AppsCode
    KubeVault
    github-icon twitter-icon
    TRY NOW FREE

    New to KubeVault? Please start here.

    Manage etcd credentials using the KubeVault operator

    OpenBao’s etcd-database-plugin is a dynamic-credentials database plugin for etcd, built on etcd’s built-in v3 Auth API via the official go.etcd.io/etcd/client/v3 client. Each issued credential becomes a native etcd user created via UserAdd. The plugin can grant pre-existing roles listed under roles and can create inline roles with key permissions defined under custom_roles before granting them to the user.

    The same CRD shape is used both for the in-process etcd-database-plugin and for the hub-spoke remote-etcd-plugin; the difference is whether the Vault AppBinding referenced by SecretEngine.spec.vaultRef is marked deploymentMode: RemoteAgent (then the SecretEngine controller rewrites plugin_name to remote-etcd-plugin and attaches spoke_name).

    You need to be familiar with the following CRDs:

    Before you begin

    • Install KubeVault operator in your cluster from here.
    • Run an etcd cluster with authentication enabled (etcdctl auth enable). The root user configured on the AppBinding must be able to add users, create roles, grant role permissions, and grant roles to users.
    • Pre-create roles referenced under roles (for example, reader or writer via etcdctl role add). Roles defined under custom_roles do not need to exist beforehand; the plugin creates them if necessary.
    $ kubectl create ns demo
namespace/demo created

    Vault Server

    Deploy a Vault Server using the KubeVault operator: Deploy Vault Server. The KubeVault operator will create an AppBinding wiring up Kubernetes auth.

    $ kubectl get appbinding -n demo vault -o yaml

    AppBinding for etcd

    Create an AppBinding pointing at the etcd client endpoint. The referenced Secret carries the root username and password used to authenticate against etcd’s v3 Auth API when the plugin manages users, roles, role permissions, and role grants.

    apiVersion: appcatalog.appscode.com/v1alpha1
kind: AppBinding
metadata:
  name: etcd
  namespace: demo
spec:
  clientConfig:
    url: http://etcd.demo.svc:2379
  secret:
    kind: Secret
    name: etcd-cred
---
apiVersion: v1
kind: Secret
metadata:
  name: etcd-cred
  namespace: demo
type: kubernetes.io/basic-auth
stringData:
  username: root
  password: change-me

    TLS Configuration via AppBinding: If the etcd cluster serves client traffic over TLS, configure it in the etcd AppBinding via spec.clientConfig.caBundle or spec.tlsSecret (and optionally spec.clientConfig.insecureSkipTLSVerify for self-signed certificates). The KubeVault operator automatically detects TLS from the referenced AppBinding and configures TLS (use_tls, CA certificate, and verification flags) for the etcd plugin. Do not specify useTLS or insecure on the SecretEngine object — etcd has no such fields.

    Enable and Configure etcd Secret Engine

    When a SecretEngine crd object is created, the KubeVault operator will enable a secret engine on a specified path and configure the secret engine with the given configuration.

    A sample SecretEngine for etcd:

    apiVersion: engine.kubevault.com/v1alpha1
kind: SecretEngine
metadata:
  name: etcd-engine
  namespace: demo
spec:
  vaultRef:
    name: vault
  etcd:
    databaseRef:
      name: etcd
      namespace: demo
    pluginName: etcd-database-plugin   # optional; this is the default
    allowedRoles:
      - "*"

    Apply it and wait for STATUS=Success:

    $ kubectl apply -f etcd-engine.yaml
secretengine.engine.kubevault.com/etcd-engine created

$ kubectl get secretengines -n demo
NAME          STATUS    AGE
etcd-engine   Success   10s

    Use kubectl describe secretengine -n demo etcd-engine to inspect error events, if any.

    Behind the scenes the KubeVault operator writes:

    bao write database/config/k8s.<cluster>.demo.etcd \
    plugin_name=etcd-database-plugin \
    endpoints=http://etcd.demo.svc:2379 \
    username=root \
    password=change-me \
    allowed_roles="*"

    Create an EtcdRole

    An EtcdRole describes how the plugin should mint a dynamic credential. creationStatements is a single-element string slice holding a JSON role document. You can name pre-existing roles under roles and/or define inline custom roles under custom_roles with specific key permissions. When custom roles are provided, the plugin creates missing roles and grants or updates the listed permissions.

    Custom roles are additive-only. Updating creationStatements does not revoke permissions omitted from the new definition, and changing a key or range leaves the old range in place. To narrow access, revoke obsolete permissions directly in etcd or migrate to a new, uniquely named role and retire the old role after its active credentials are revoked. Avoid sharing custom-role names between EtcdRole objects unless accumulated permissions are intentional.

    Example with a pre-existing role:

    apiVersion: engine.kubevault.com/v1alpha1
kind: EtcdRole
metadata:
  name: etcd-reader
  namespace: demo
spec:
  secretEngineRef:
    name: etcd-engine
  creationStatements:
    - '{"roles":["reader"]}'
  defaultTTL: 1h
  maxTTL: 24h

    Apply and verify:

    $ kubectl apply -f etcd-role.yaml
etcdrole.engine.kubevault.com/etcd-reader created

$ kubectl get etcdrole -n demo
NAME          STATUS    AGE
etcd-reader   Success   12s

    The role name in Vault follows the format k8s.{clusterName}.{metadata.namespace}.{metadata.name}, so you can verify directly with the Vault CLI:

    $ vault read your-database-path/roles/k8s.-.demo.etcd-reader
Key                      Value
---                      -----
creation_statements      [{"roles":["reader"]}]
db_name                  k8s.-.demo.etcd
default_ttl              1h
max_ttl                  24h

    Example with inline custom_roles:

    apiVersion: engine.kubevault.com/v1alpha1
kind: EtcdRole
metadata:
  name: etcd-app-writer
  namespace: demo
spec:
  secretEngineRef:
    name: etcd-engine
  creationStatements:
    - |
      {
        "roles": ["reader"],
        "custom_roles": [
          {
            "name": "app_writer",
            "permissions": [
              {
                "permission": "readwrite",
                "key": "/app/",
                "prefix": true
              }
            ]
          }
        ]
      }      
  defaultTTL: 1h
  maxTTL: 24h

    Deleting the EtcdRole removes the role from Vault. Ephemeral users issued from this role will be revoked without deleting the custom role or its accumulated permissions from the etcd cluster.

    Issue etcd credentials

    Request a dynamic credential by creating a SecretAccessRequest:

    apiVersion: engine.kubevault.com/v1alpha1
kind: SecretAccessRequest
metadata:
  name: etcd-cred-rqst
  namespace: demo
spec:
  roleRef:
    kind: EtcdRole
    name: etcd-reader
  subjects:
    - kind: ServiceAccount
      name: demo-sa
      namespace: demo

    Approve it through the KubeVault CLI:

    $ kubectl vault approve secretaccessrequest etcd-cred-rqst -n demo
approved

    Once approved, the operator issues the credential, stores it in a Secret, and binds the listed subjects via a Role/RoleBinding. The plugin creates a new etcd user via UserAdd and grants it the reader role via UserGrantRole. The credential lives on the lease until you delete the SecretAccessRequest or it expires; on lease revocation the plugin deletes the etcd user via UserDelete (treating a not-found user as success, so revocation is idempotent).

    $ kubectl get secretaccessrequest etcd-cred-rqst -n demo -o json | jq '.status'
{
  "lease": {
    "duration": "1h0m0s",
    "id": "your-database-path/creds/k8s.-.demo.etcd-reader/abc...",
    "renewable": true
  },
  "secret": {
    "name": "etcd-cred-rqst-xxxxxx"
  }
}

$ kubectl get secret -n demo etcd-cred-rqst-xxxxxx -o jsonpath='{.data.username}' | base64 -d
v-kubernetes-demo-XXXXXXXX

$ kubectl get secret -n demo etcd-cred-rqst-xxxxxx -o jsonpath='{.data.password}' | base64 -d
xxxxxxxxxxxxxxxxxx

    Use the issued username / password as your etcd client’s auth credentials (e.g. etcdctl --user username:password); the credential is revoked when the SecretAccessRequest is deleted.

    Further reading

    What's on this Pages

    Search
    Improve This Page