Mount PKI(certificates) Secrets using CSI Driver

Kubernetes Secrets Store CSI Driver

Secrets Store CSI driver for Kubernetes secrets - Integrates secrets stores with Kubernetes via a Container Storage Interface (CSI) volume.

The Secrets Store CSI driver secrets-store.csi.k8s.io allows Kubernetes to mount multiple secrets, keys, and certs stored in enterprise-grade external secrets stores into their pods as a volume. Once the Volume is attached, the data in it is mounted into the container’s file system.

Secrets-store CSI architecture

When the Pod is created through the K8s API, it’s scheduled on to a node. The kubelet process on the node looks at the pod spec & see if there’s any volumeMount request. The kubelet issues an RPC to the CSI driver to mount the volume. The CSI driver creates & mounts tmpfs into the pod. Then the CSI driver issues a request to the Provider. The provider talks to the external secrets store to fetch the secrets & write them to the pod volume as files. At this point, volume is successfully mounted & the pod starts running.

You can read more about the Kubernetes Secrets Store CSI Driver here.

Consuming Secrets

At first, you need to have a Kubernetes 1.16 or later cluster, and the kubectl command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using kind. To check the version of your cluster, run:

$ kubectl version --short
Client Version: v1.21.2
Server Version: v1.21.1

Before you begin:

  • Install KubeVault operator in your cluster from here.
  • Install Secrets Store CSI driver for Kubernetes secrets in your cluster from here.

To keep things isolated, we are going to use a separate namespace called demo throughout this tutorial.

$ kubectl create ns demo
namespace/demo created

Note: YAML files used in this tutorial stored in examples folder in GitHub repository KubeVault/docs

Vault Server

If you don’t have a Vault Server, you can deploy it by using the KubeVault operator.

The KubeVault operator can manage policies and secret engines of Vault servers which are not provisioned by the KubeVault operator. You need to configure both the Vault server and the cluster so that the KubeVault operator can communicate with your Vault server.

Now, we have the AppBinding that contains connection and authentication information about the Vault server. And we also have the service account that the Vault server can authenticate.

$ kubectl get appbinding -n demo
NAME    AGE
vault   50m

$ kubectl get appbinding -n demo vault -o yaml
apiVersion: appcatalog.appscode.com/v1alpha1
kind: AppBinding
metadata:
  creationTimestamp: "2021-08-16T08:23:38Z"
  generation: 1
  labels:
    app.kubernetes.io/instance: vault
    app.kubernetes.io/managed-by: kubevault.com
    app.kubernetes.io/name: vaultservers.kubevault.com
  name: vault
  namespace: demo
  ownerReferences:
  - apiVersion: kubevault.com/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: VaultServer
    name: vault
    uid: 6b405147-93da-41ff-aad3-29ae9f415d0a
  resourceVersion: "602898"
  uid: b54873fd-0f34-42f7-bdf3-4e667edb4659
spec:
  clientConfig:
    service:
      name: vault
      port: 8200
      scheme: http
  parameters:
    apiVersion: config.kubevault.com/v1alpha1
    kind: VaultServerConfiguration
    kubernetes:
      serviceAccountName: vault
      tokenReviewerServiceAccountName: vault-k8s-token-reviewer
      usePodServiceAccountForCSIDriver: true
    path: kubernetes
    vaultRole: vault-policy-controller

Enable and Configure PKI Secret Engine

We will use the Vault CLI throughout the tutorial to enable and configure the PKI secret engine.

Don’t have Vault CLI? Download and configure it as described here

To use secret from PKI secret engine, you have to perform the following steps.

Enable PKI Secret Engine

To enable PKI secret engine run the following command.

$ vault secrets enable pki
Success! Enabled the pki secrets engine at: pki/

Increase the TTL by tuning the secrets engine. The default value of 30 days may be too short, so increase it to 1 year:

$ vault secrets tune -max-lease-ttl=8760h pki
Success! Tuned the secrets engine at: pki/

Configure CA Certificate and Private Key

Configure a CA certificate and private key. Vault can accept an existing key pair, or it can generate its own self-signed root.

$ vault write pki/root/generate/internal \
                          common_name=my-website.com \
                          ttl=8760h
Key              Value
---              -----
certificate      -----BEGIN CERTIFICATE-----
MIIDPjCCAiagAwIBAgIUEDmnAmC0siISlrezD3/CeUXTSfswDQYJKoZIhvcNAQEL
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
CsFVu+vfMM9XEMYeKHRWAq9onJFyGuwKGhF0/7RbZ3EunTj6Zph+UMucGoL4xfXj
ITltdU1N4JPvihQq+8Omryay
-----END CERTIFICATE-----
expiration       1606200496
issuing_ca       -----BEGIN CERTIFICATE-----
MIIDPjCCAiagAwIBAgIUEDmnAmC0siISlrezD3/CeUXTSfswDQYJKoZIhvcNAQEL
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
CsFVu+vfMM9XEMYeKHRWAq9onJFyGuwKGhF0/7RbZ3EunTj6Zph+UMucGoL4xfXj
ITltdU1N4JPvihQq+8Omryay
-----END CERTIFICATE-----
serial_number    10:39:a7:02:60:b4:b2:22:12:96:b7:b3:0f:7f:c2:79:45:d3:49:fb

Configure a PKI Role

We need to configure a role that maps a name in vault to a procedure for generating certificate. When users of machines generate credentials, they are generated agains this role:

$ vault write pki/roles/example-dot-com \
                          allowed_domains=my-website.com \
                          allow_subdomains=true \
                          max_ttl=72h
Success! Data written to: pki/roles/example-dot-com

Create Service Account for Pod

Let’s create the service account test-user-account which will be used in VaultPolicyBinding.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: test-user-account
  namespace: demo
$ kubectl apply -f docs/examples/guides/secret-engines/pki/serviceaccount.yaml
serviceaccount/test-user-account created

$ kubectl get serviceaccount -n demo
NAME                SECRETS   AGE
test-user-account   1         4h10m

Create VaultPolicy and VaultPolicyBinding for Pod’s Service Account

When a VaultPolicyBinding object is created, the KubeVault operator create an auth role in the Vault server. The role name is generated by the following naming format: k8s.(clusterName or -).namespace.name. Here, it is k8s.-.demo.pki-se-role.

apiVersion: policy.kubevault.com/v1alpha1
kind: VaultPolicy
metadata:
  name: pki-se-policy
  namespace: demo
spec:
  vaultRef:
    name: vault
  policyDocument: |
    path "pki/issue/*" {
      capabilities = ["update"]
    }    
---
apiVersion: policy.kubevault.com/v1alpha1
kind: VaultPolicyBinding
metadata:
  name: pki-se-role
  namespace: demo
spec:
  vaultRef:
    name: vault
  policies:
    - ref: pki-se-policy
  subjectRef:
    kubernetes:
      serviceAccountNames:
        - "test-user-account"
      serviceAccountNamespaces:
        - "demo"

Let’s create VaultPolicy and VaultPolicyBinding:

$ kubectl apply -f docs/examples/guides/secret-engines/pki/policy.yaml
vaultpolicy.policy.kubevault.com/pki-se-policy created

$ kubectl apply -f docs/examples/guides/secret-engines/pki/policybinding.yaml
vaultpolicybinding.policy.kubevault.com/pki-se-role created

Check if the VaultPolicy and the VaultPolicyBinding are successfully registered to the Vault server:

$ kubectl get vaultpolicy -n demo
NAME                           STATUS    AGE
pki-se-policy                  Success   8s

$ kubectl get vaultpolicybinding -n demo
NAME                          STATUS    AGE
pki-se-role                   Success   10s

Mount Certificates into a Kubernetes Pod

So, we can create SecretProviderClass now. You can read more about SecretProviderClass here.

Create SecretProviderClass

Create SecretProviderClass object with the following content:

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: vault-db-provider
  namespace: demo
spec:
  provider: vault
  parameters:
    vaultAddress: "http://vault.demo:8200"
    roleName: "k8s.-.demo.pki-se-role"
    objects: |
      - objectName: "certificate"
        secretPath: "pki/issue/example-dot-com"
        secretKey: "certificate"
        secretArgs:
          common_name: "www.my-website.com"
          ttl: 24h
        method: "POST"

      - objectName: "issuing_ca"
        secretPath: "pki/issue/example-dot-com"
        secretKey: "issuing_ca"
        secretArgs:
          common_name: "www.my-website.com"
          ttl: 24h
        method: "POST"

      - objectName: "private_key"
        secretPath: "pki/issue/example-dot-com"
        secretKey: "private_key"
        secretArgs:
          common_name: "www.my-website.com"
          ttl: 24h
        method: "POST"

      - objectName: "private_key_type"
        secretPath: "pki/issue/example-dot-com"
        secretKey: "private_key_type"
        secretArgs:
          common_name: "www.my-website.com"
          ttl: 24h
        method: "POST"      
$ kubectl apply -f docs/examples/guides/secret-engines/pki/secretproviderclass.yaml
secretproviderclass.secrets-store.csi.x-k8s.io/vault-db-provider created

Here, you can also pass the following parameters optionally to issue the certificate

  • common_name (string: ) – Specifies the requested CN for the certificate. If the CN is allowed by role policy, it will be issued.

  • alt_names (string: “”) – Specifies requested Subject Alternative Names, in a comma-delimited list. These can be host names or email addresses; they will be parsed into their respective fields. If any requested names do not match role policy, the entire request will be denied.

  • ip_sans (string: “”) – Specifies requested IP Subject Alternative Names, in a comma-delimited list. Only valid if the role allows IP SANs (which is the default).

  • uri_sans (string: “”) – Specifies the requested URI Subject Alternative Names, in a comma-delimited list.

  • other_sans (string: “”) – Specifies custom OID/UTF8-string SANs. These must match values specified on the role in allowed_other_sans (globbing allowed). The format is the same as OpenSSL: ;: where the only current valid type is UTF8. This can be a comma-delimited list or a JSON string slice.

  • ttl (string: “”) – Specifies requested Time To Live. Cannot be greater than the role’s max_ttl value. If not provided, the role’s ttl value will be used. Note that the role values default to system values if not explicitly set.

  • format (string: “”) – Specifies the format for returned data. Can be pem, der, or pem_bundle; defaults to pem. If der, the output is base64 encoded. If pem_bundle, the certificate field will contain the private key and certificate, concatenated; if the issuing CA is not a Vault-derived self-signed root, this will be included as well.

  • private_key_format (string: “”) – Specifies the format for marshaling the private key. Defaults to der which will return either base64-encoded DER or PEM-encoded DER, depending on the value of format. The other option is pkcs8 which will return the key marshalled as PEM-encoded PKCS8.

  • exclude_cn_from_sans (bool: false) – If true, the given common_name will not be included in DNS or Email Subject Alternate Names (as appropriate). Useful if the CN is not a hostname or email address, but is instead some human-readable identifier.

NOTE: The SecretProviderClass needs to be created in the same namespace as the pod.

Create Pod

Now we can create a Pod to consume the PKI secrets. When the Pod is created, the Provider fetches the secret and writes them to Pod’s volume as files. At this point, the volume is successfully mounted and the Pod starts running.

apiVersion: v1
kind: Pod
metadata:
  name: demo-app
  namespace: demo
spec:
  serviceAccountName: test-user-account
  containers:
    - image: jweissig/app:0.0.1
      name: demo-app
      imagePullPolicy: Always
      volumeMounts:
        - name: secrets-store-inline
          mountPath: "/secrets-store/pki-assets"
          readOnly: true
  volumes:
    - name: secrets-store-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "vault-db-provider"
$ kubectl apply -f docs/examples/guides/secret-engines/pki/pod.yaml
pod/demo-app created

Test & Verify

Check if the Pod is running successfully, by running:

$ kubectl get pods -n demo
NAME                       READY   STATUS    RESTARTS   AGE
demo-app                   1/1     Running   0          11s

Verify Secret

If the Pod is running successfully, then check inside the app container by running

$ kubectl exec -it -n test pod/demo-app -- /bin/sh

/ # ls /secrets-store/pki-assets
certificate       issuing_ca        private_key       private_key_type

/ # cat /secrets-store/pki-assets/certificate
-----BEGIN CERTIFICATE-----
MIIDVjCCAj6gAwIBAgIUNjTBC3qR7Zaj0XrzUc3QEbE+EhgwDQYJKoZIhvcNAQEL
BQAwGTEXMBUGA1UEAxMObXktd2Vic2l0ZS5jb20wHhcNMTkxMjEzMTExNDIwWhcN
..... .... .... .... .... .... .... .... .... .... .... .... ...
bo901cITjNyCTbAF2401pYFZ4rSlxhcuAvc7c27uqvKEh2/ctMGRkvPVygbPdvB8
LfCskfX0sk8PQiEznlmYlChK3KNsEp+xSCyjU+pDEw8AcDXwE6vVFft/fRX0oiHH
KIzTZ7R/QKUkLisloMUHStINISAehglLZTJjo79jB7GN66wyqP+E8iRLEYFAAsb0
aZ5wuSTYEpqOuP6G1tOdhiE7iptFu9Wg9dKtmXkZnc0iTBL60xMUUapH
-----END CERTIFICATE-----

Cleaning up

To clean up the Kubernetes resources created by this tutorial, run:

$ kubectl delete ns demo
namespace "demo" deleted