You are looking at the documentation of a prior release. To read the documentation of the latest release, please visit here.

Mount MySQL Secrets using CSI Driver

Kubernetes Secrets Store CSI Driver

Secrets Store CSI driver for Kubernetes secrets - Integrates secrets stores with Kubernetes via a Container Storage Interface (CSI) volume.

The Secrets Store CSI driver allows Kubernetes to mount multiple secrets, keys, and certs stored in enterprise-grade external secrets stores into their pods as a volume. Once the Volume is attached, the data in it is mounted into the container’s file system.

Secrets-store CSI architecture

When the Pod is created through the K8s API, it’s scheduled on to a node. The kubelet process on the node looks at the pod spec & see if there’s any volumeMount request. The kubelet issues an RPC to the CSI driver to mount the volume. The CSI driver creates & mounts tmpfs into the pod. Then the CSI driver issues a request to the Provider. The provider talks to the external secrets store to fetch the secrets & write them to the pod volume as files. At this point, volume is successfully mounted & the pod starts running.

You can read more about the Kubernetes Secrets Store CSI Driver here.

Consuming Secrets

At first, you need to have a Kubernetes 1.16 or later cluster, and the kubectl command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using kind. To check the version of your cluster, run:

$ kubectl version --short
Client Version: v1.21.2
Server Version: v1.21.1

Before you begin:

  • Install KubeVault operator in your cluster from here.
  • Install Secrets Store CSI driver for Kubernetes secrets in your cluster from here.
  • Install Vault Specific CSI provider from here

To keep things isolated, we are going to use a separate namespace called demo throughout this tutorial.

$ kubectl create ns demo
namespace/demo created

Note: YAML files used in this tutorial stored in examples folder in GitHub repository KubeVault/docs

Vault Server

If you don’t have a Vault Server, you can deploy it by using the KubeVault operator.

The KubeVault operator can manage policies and secret engines of Vault servers which are not provisioned by the KubeVault operator. You need to configure both the Vault server and the cluster so that the KubeVault operator can communicate with your Vault server.

Now, we have the AppBinding that contains connection and authentication information about the Vault server. And we also have the service account that the Vault server can authenticate.

$ kubectl get appbinding -n demo
vault   50m

$ kubectl get appbinding -n demo vault -o yaml
kind: AppBinding
  creationTimestamp: "2021-08-16T08:23:38Z"
  generation: 1
  labels: vault
  name: vault
  namespace: demo
  - apiVersion:
    blockOwnerDeletion: true
    controller: true
    kind: VaultServer
    name: vault
    uid: 6b405147-93da-41ff-aad3-29ae9f415d0a
  resourceVersion: "602898"
  uid: b54873fd-0f34-42f7-bdf3-4e667edb4659
      name: vault
      port: 8200
      scheme: http
    kind: VaultServerConfiguration
      serviceAccountName: vault
      tokenReviewerServiceAccountName: vault-k8s-token-reviewer
      usePodServiceAccountForCSIDriver: true
    path: kubernetes
    vaultRole: vault-policy-controller

Enable & Configure MySQL SecretEngine

Enable MySQL SecretEngine

$ kubectl apply -f docs/examples/guides/secret-engines/mysql/secretengine.yaml created

Create MySQLRole

$ kubectl apply -f docs/examples/guides/secret-engines/mysql/secretenginerole.yaml created

Let’s say pod’s service account name is test-user-account located in demo namespace. We need to create a VaultPolicy and a VaultPolicyBinding so that the pod has access to read secrets from the Vault server.

Create Service Account for Pod

Let’s create the service account test-user-account which will be used in VaultPolicyBinding.

apiVersion: v1
kind: ServiceAccount
  name: test-user-account
  namespace: demo
$ kubectl apply -f docs/examples/guides/secret-engines/mysql/serviceaccount.yaml
serviceaccount/test-user-account created

$ kubectl get serviceaccount -n demo
NAME                SECRETS   AGE
test-user-account   1         4h10m

Create VaultPolicy and VaultPolicyBinding for Pod’s Service Account

When a VaultPolicyBinding object is created, the KubeVault operator create an auth role in the Vault server. The role name is generated by the following naming format: k8s.(clusterName or -) Here, it is k8s.-.demo.mysql-reader-role.

kind: VaultPolicy
  name: mysql-reader-policy
  namespace: demo
    name: vault
  policyDocument: |
    path "your-database-path/creds/k8s.-.demo.mysql-superuser-role" {
      capabilities = ["read"]
kind: VaultPolicyBinding
  name: mysql-reader-role
  namespace: demo
    name: vault
    - ref: mysql-reader-policy
        - "test-user-account"
        - "demo"

Let’s create VaultPolicy and VaultPolicyBinding:

$ kubectl apply -f docs/examples/guides/secret-engines/mysql/policy.yaml created

$ kubectl apply -f docs/examples/guides/secret-engines/mysql/policybinding.yaml created

Check if the VaultPolicy and the VaultPolicyBinding are successfully registered to the Vault server:

$ kubectl get vaultpolicy -n demo
NAME                                 STATUS    AGE
mysql-reader-policy                  Success   8s

$ kubectl get vaultpolicybinding -n demo
NAME                                 STATUS    AGE
mysql-reader-role                    Success   10s

Mount secrets into a Kubernetes pod

So, we can create SecretProviderClass now. You can read more about SecretProviderClass here.

Create SecretProviderClass

Create SecretProviderClass object with the following content:

kind: SecretProviderClass
  name: vault-db-provider
  namespace: demo
  provider: vault
    vaultAddress: "http://vault.demo:8200"
    roleName: "k8s.-.demo.mysql-reader-role"
    objects: |
      - objectName: "mysql-creds-username"
        secretPath: "your-database-path/creds/k8s.-.demo.mysql-superuser-role"
        secretKey: "username"
      - objectName: "mysql-creds-password"
        secretPath: "your-database-path/creds/k8s.-.demo.mysql-superuser-role"
        secretKey: "password"      
$ kubectl apply -f docs/examples/guides/secret-engines/mysql/secretproviderclass.yaml created

NOTE: The SecretProviderClass needs to be created in the same namespace as the pod.

Create Pod

Now we can create a Pod to consume the MySQL secrets. When the Pod is created, the Provider fetches the secret and writes them to Pod’s volume as files. At this point, the volume is successfully mounted and the Pod starts running.

apiVersion: v1
kind: Pod
  name: demo-app
  namespace: demo
  serviceAccountName: test-user-account
    - image: jweissig/app:0.0.1
      name: demo-app
      imagePullPolicy: Always
        - name: secrets-store-inline
          mountPath: "/secrets-store/mysql-creds"
          readOnly: true
    - name: secrets-store-inline
        readOnly: true
          secretProviderClass: "vault-db-provider"
$ kubectl apply -f docs/examples/guides/secret-engines/mysql/pod.yaml
pod/demo-app created

Test & Verify

Check if the Pod is running successfully, by running:

$ kubectl get pods -n demo
NAME                       READY   STATUS    RESTARTS   AGE
demo-app                   1/1     Running   0          11s

Verify Secret

If the Pod is running successfully, then check inside the app container by running

$ kubectl exec -it -n demo pod/demo-app -- /bin/sh
/ # ls /secrets-store/mysql-creds
mysql-creds-password  mysql-creds-username

/ # cat /secrets-store/mysql-creds/mysql-creds-password

/ # cat /secrets-store/mysql-creds/mysql-creds-username

/ # exit

So, we can see that the secret db-username and db-password is mounted into the pod, where the secret key is mounted as file and value is the content of that file.

Cleaning up

To clean up the Kubernetes resources created by this tutorial, run:

$ kubectl delete ns demo
namespace "demo" deleted