New to KubeVault? Please start here.

Manage PKI(certificates) secrets using the KubeVault operator

The PKI secrets engine generates dynamic X.509 certificates. With this secrets engine, services can get certificates without going through the usual manual process of generating a private key and CSR, submitting to a CA, and waiting for a verification and signing process to complete. Vault’s built-in authentication and authorization mechanisms provide the verification functionality.

You can easily manage PKI secret engine using the KubeVault operator.

You should be familiar with the following CRD:

Before you begin

  • Install KubeVault operator in your cluster from here.

To keep things isolated, we are going to use a separate namespace called demo throughout this tutorial.

$ kubectl create ns demo
namespace/demo created

In this tutorial, we are going to demonstrate the use of the PKI secret engine.

Vault Server

If you don’t have a Vault Server, you can deploy it by using the KubeVault operator.

The KubeVault operator can manage policies and secret engines of Vault servers which are not provisioned by the KubeVault operator. You need to configure both the Vault server and the cluster so that the KubeVault operator can communicate with your Vault server.

Now, we have the AppBinding that contains connection and authentication information about the Vault server.

$ kubectl get appbinding -n demo
NAME    AGE
vault   50m

$ kubectl get appbinding -n demo vault -o yaml
apiVersion: appcatalog.appscode.com/v1alpha1
kind: AppBinding
metadata:
  creationTimestamp: "2021-08-16T08:23:38Z"
  generation: 1
  labels:
    app.kubernetes.io/instance: vault
    app.kubernetes.io/managed-by: kubevault.com
    app.kubernetes.io/name: vaultservers.kubevault.com
  name: vault
  namespace: demo
  ownerReferences:
  - apiVersion: kubevault.com/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: VaultServer
    name: vault
    uid: 6b405147-93da-41ff-aad3-29ae9f415d0a
  resourceVersion: "602898"
  uid: b54873fd-0f34-42f7-bdf3-4e667edb4659
spec:
  clientConfig:
    service:
      name: vault
      port: 8200
      scheme: http
  parameters:
    apiVersion: config.kubevault.com/v1alpha1
    kind: VaultServerConfiguration
    kubernetes:
      serviceAccountName: vault
      tokenReviewerServiceAccountName: vault-k8s-token-reviewer
      usePodServiceAccountForCSIDriver: true
    path: kubernetes
    vaultRole: vault-policy-controller

Use PKI Secret Engine as Root User

Here, we are going to use the Vault root token to perform authentication to the Vault server. We will use the Vault CLI throughout the tutorial.

Don’t have Vault CLI? Download and configure it as described here

Export the root token as environment variable:

$ export VAULT_TOKEN=s.diWLjSzmfSmF0qUNYV3qOIeX

Enable the PKI secrets engine:

$ vault secrets enable pki
Success! Enabled the pki secrets engine at: pki/

Increase the TTL by tuning the secrets engine. The default value of 30 days may be too short, so increase it to 1 year:

$ vault secrets tune -max-lease-ttl=8760h pki
Success! Tuned the secrets engine at: pki/

Configure a CA certificate and private key:

$ vault write pki/root/generate/internal \
                          common_name=my-website.com \
                          ttl=8760h
Key              Value
---              -----
certificate      -----BEGIN CERTIFICATE-----
MIIDPjCCAiagAwIBAgIUEDmnAmC0siISlrezD3/CeUXTSfswDQYJKoZIhvcNAQEL
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
CsFVu+vfMM9XEMYeKHRWAq9onJFyGuwKGhF0/7RbZ3EunTj6Zph+UMucGoL4xfXj
ITltdU1N4JPvihQq+8Omryay
-----END CERTIFICATE-----
expiration       1606200496
issuing_ca       -----BEGIN CERTIFICATE-----
MIIDPjCCAiagAwIBAgIUEDmnAmC0siISlrezD3/CeUXTSfswDQYJKoZIhvcNAQEL
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
CsFVu+vfMM9XEMYeKHRWAq9onJFyGuwKGhF0/7RbZ3EunTj6Zph+UMucGoL4xfXj
ITltdU1N4JPvihQq+8Omryay
-----END CERTIFICATE-----
serial_number    10:39:a7:02:60:b4:b2:22:12:96:b7:b3:0f:7f:c2:79:45:d3:49:fb

Configure a role that maps a name in Vault to a procedure for generating a certificate. When users or machines generate credentials, they are generated against this role:

$ vault write pki/roles/example-dot-com \
                          allowed_domains=my-website.com \
                          allow_subdomains=true \
                          max_ttl=72h
Success! Data written to: pki/roles/example-dot-com

Generate a new credential by writing to the /issue endpoint with the name of the role:

$ vault write pki/issue/example-dot-com \
                        common_name=www.my-website.com
Key                 Value
---                 -----
certificate         -----BEGIN CERTIFICATE-----
MIIDVjCCAj6gAwIBAgIUWQhPLW6R/nk/3x3XReHC1Ze4BWUwDQYJKoZIhvcNAQEL
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
TSuguIiSBt5NN0ou4aY01FbeJJOHZhtpj31XdXOCAKR40lPCmWtEUAbcuEhLlkm+
vmhNYxBqkx33jEIMxk95P4eKIYPyr45/8o7bV1jq7G26aBzj1Mjd0JmU
-----END CERTIFICATE-----
expiration          1574924103
issuing_ca          -----BEGIN CERTIFICATE-----
MIIDPjCCAiagAwIBAgIUEDmnAmC0siISlrezD3/CeUXTSfswDQYJKoZIhvcNAQEL
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
CsFVu+vfMM9XEMYeKHRWAq9onJFyGuwKGhF0/7RbZ3EunTj6Zph+UMucGoL4xfXj
ITltdU1N4JPvihQq+8Omryay
-----END CERTIFICATE-----
private_key         -----BEGIN RSA PRIVATE KEY-----
MIIEpAIBAAKCAQEAuK7V4GuoHSF8pnlr4hApeU7V3zpuQ2rWt3pXgi9TPBCmIuye
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
+o8HetGW5xWvuQ/ObkiSzdQ8nxMyiQj/whe4riYriOw1fYwPrjZfxTm1jsyEmbbm
gYewhfHP3hOgTCVu3SjhvOXS3pnW7hUP4wtvpLLdRumEUM/fK7pwNg==
-----END RSA PRIVATE KEY-----
private_key_type    rsa
serial_number       59:008:4f:2d:6e:91:fe:79:3f:df:1d:d7:45:e1:c2:d5:97:b8:05:65

For more details visit the official Vault documentation.

Use PKI Secret Engine as Non-root User

Here, we are going to create a Kubernetes service account and give it limited access (i.e only PKI secret engine) from the Vault using the VaultPolicy and the VaultPolicyBinding.

Create Kubernetes Service Account

Create a service account pki-admin to the demo namespace:

$  kubectl create serviceaccount -n demo pki-admin
serviceaccount/pki-admin created

# get service account JWT token which will be required while performing
# login operation to the Vault
$ kubectl get secrets -n demo pki-admin-token-26kwb  -o jsonpath="{.data.token}" | base64 --decode;
eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZW1vIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InBraS1hZG1pbi10b2tlbi0yNmt3YiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJwa2ktYWRtaW4iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJkYmVkZDQ2Ni0yYzc0LTQ0OGItOTBlZS01MDlkNGI4MTJjOTEiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVtbzpwa2ktYWRtaW4ifQ.ce7OqA05nsfBMRsEOiG1Lje_mOBdUZRKALB9Sc9LVqjKIJZHdxvZ7NT4ZKrIyPEe02aItzxlXLAP4Fa8dUMshZuNyuxBYN7p2qHRCwVKHqOuz8LdRQWypKiLozL9v0DHk-vbFWFcm0eye57vJBFtriYyYRUA84WZhxRb9wz-f8z7PSmO2mpjkrICt7wi48j-4FObdhFWk6HAKXFD7bCzL4j3CWUcx2wTIsnOEz9SifjYZuGaog6tpWhnj-guEKpXJzBLAoMBU0Vr3U7Zv_z1qvKFF4ZherUBxSOMo27lL2xbhkpbW2wf_DCAjLx8pScoh9mxv7AK2WJCHeA0JRzrug

Create VaultPolicy and VaultPolicyBinding

A sample VaultPolicy object with necessary path permission for the PKI secret engine:

apiVersion: policy.kubevault.com/v1alpha1
kind: VaultPolicy
metadata:
  name: pki-policy
  namespace: demo
spec:
  vaultRef:
    name: vault
  policyDocument: |
    path "sys/mounts" {
      capabilities = ["read", "list"]
    }

    path "sys/mounts/*" {
      capabilities = ["create", "read", "update", "delete"]
    }

    path "pki/*" {
        capabilities = ["read","create", "list", "update", "delete"]
    }

    path "sys/leases/revoke/*" {
        capabilities = ["update"]
    }    

Create VaultPolicy and check status:

$ kubectl apply -f docs/examples/guides/secret-engines/pki/policy.yaml
vaultpolicy.policy.kubevault.com/pki-policy created

$ kubectl get vaultpolicy -n demo
NAME                           STATUS    AGE
pki-policy                     Success   3m15s

A sample VaultPolicyBinding object that binds the pki-policy to the pki-admin service account:

apiVersion: policy.kubevault.com/v1alpha1
kind: VaultPolicyBinding
metadata:
  name: pki-admin-role
  namespace: demo
spec:
  vaultRef:
    name: vault
  policies:
  - ref: pki-policy
  subjectRef:
    kubernetes:
      serviceAccountNames:
        - "pki-admin"
      serviceAccountNamespaces:
        - "demo"
      ttl: "1000"
      maxTTL: "2000"
      period: "1000"

Create VaultPolicyBinding and check status:

$ kubectl apply -f docs/examples/guides/secret-engines/pki/policyBinding.yaml
vaultpolicybinding.policy.kubevault.com/pki-admin-role created

$ kubectl get vaultpolicybindings -n demo
NAME                           STATUS    AGE
pki-admin-role                 Success   43m

Login Vault and Use PKI Secret Engine

To resolve the naming conflict, name of the policy and role in Vault will follow this format: k8s.{clusterName}.{metadata.namespace}.{metadata.name}.

Don’t have Vault CLI? Download and configure it as described here

List Vault policies and Kubernetes auth roles:

$ vault list sys/policy
Keys
----
k8s.-.demo.pki-policy

$ vault read sys/policy/k8s.-.demo.pki-policy
Key      Value
---      -----
name     k8s.-.demo.pki-policy
rules    path "sys/mounts" {
  capabilities = ["read", "list"]
}

path "sys/mounts/*" {
  capabilities = ["create", "read", "update", "delete"]
}

path "pki/*" {
    capabilities = ["read","create", "list", "update", "delete"]
}

path "sys/leases/revoke/*" {
    capabilities = ["update"]
}

$ vault list auth/kubernetes/role
Keys
----
k8s.-.demo.pki-admin-role

$ vault read auth/kubernetes/role/k8s.-.demo.pki-admin-role
Key                                 Value
---                                 -----
bound_service_account_names         [pki-admin]
bound_service_account_namespaces    [demo]
max_ttl                             33m20s
period                              16m40s
policies                            [k8s.-.demo.pki-policy]
token_bound_cidrs                   []
token_explicit_max_ttl              0s
token_max_ttl                       33m20s
token_no_default_policy             false
token_num_uses                      0
token_period                        16m40s
token_policies                      [k8s.-.demo.pki-policy]
token_ttl                           16m40s
token_type                          default
ttl                                 16m40s

So, we can see that the pki-policy is added to the pki-admin-role.

Now, login to the Vault using pki-admin’s JWT token under pki-admin-role role.

$ vault write auth/kubernetes/login \
                       role=k8s.-.demo.pki-admin-role \
                       jwt=eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZW1vIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InBraS1hZG1pbi10b2tlbi0yNmt3YiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJwa2ktYWRtaW4iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJkYmVkZDQ2Ni0yYzc0LTQ0OGItOTBlZS01MDlkNGI4MTJjOTEiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVtbzpwa2ktYWRtaW4ifQ.ce7OqA05nsfBMRsEOiG1Lje_mOBdUZRKALB9Sc9LVqjKIJZHdxvZ7NT4ZKrIyPEe02aItzxlXLAP4Fa8dUMshZuNyuxBYN7p2qHRCwVKHqOuz8LdRQWypKiLozL9v0DHk-vbFWFcm0eye57vJBFtriYyYRUA84WZhxRb9wz-f8z7PSmO2mpjkrICt7wi48j-4FObdhFWk6HAKXFD7bCzL4j3CWUcx2wTIsnOEz9SifjYZuGaog6tpWhnj-guEKpXJzBLAoMBU0Vr3U7Zv_z1qvKFF4ZherUBxSOMo27lL2xbhkpbW2wf_DCAjLx8pScoh9mxv7AK2WJCHeA0JRzrug

Key                                       Value
---                                       -----
token                                     s.ZPu4zcyaajjpxtS1t8fnh2LV
token_accessor                            5OknOf72h8WnP1v0I1C01626
token_duration                            16m40s
token_renewable                           true
token_policies                            ["default" "k8s.-.demo.pki-policy"]
identity_policies                         []
policies                                  ["default" "k8s.-.demo.pki-policy"]
token_meta_role                           k8s.-.demo.pki-admin-role
token_meta_service_account_name           pki-admin
token_meta_service_account_namespace      demo
token_meta_service_account_secret_name    pki-admin-token-26kwb
token_meta_service_account_uid            dbedd466-2c74-448b-90ee-509d4b812c91

Export the new Vault token as an environment variable:

export VAULT_TOKEN=s.ZPu4zcyaajjpxtS1t8fnh2LV

Now generate a new certificate using the PKI secret engine:

Enable the PKI secrets engine:

$ vault secrets enable pki
Success! Enabled the pki secrets engine at: pki/

Increase the TTL by tuning the secrets engine. The default value of 30 days may be too short, so increase it to 1 year:

$ vault secrets tune -max-lease-ttl=8760h pki
Success! Tuned the secrets engine at: pki/

Configure a CA certificate and private key:

$ vault write pki/root/generate/internal \
                          common_name=my-website.com \
                          ttl=8760h
Key              Value
---              -----
certificate      -----BEGIN CERTIFICATE-----
MIIDPjCCAiagAwIBAgIUEDmnAmC0siISlrezD3/CeUXTSfswDQYJKoZIhvcNAQEL
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
CsFVu+vfMM9XEMYeKHRWAq9onJFyGuwKGhF0/7RbZ3EunTj6Zph+UMucGoL4xfXj
ITltdU1N4JPvihQq+8Omryay
-----END CERTIFICATE-----
expiration       1606200496
issuing_ca       -----BEGIN CERTIFICATE-----
MIIDPjCCAiagAwIBAgIUEDmnAmC0siISlrezD3/CeUXTSfswDQYJKoZIhvcNAQEL
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
CsFVu+vfMM9XEMYeKHRWAq9onJFyGuwKGhF0/7RbZ3EunTj6Zph+UMucGoL4xfXj
ITltdU1N4JPvihQq+8Omryay
-----END CERTIFICATE-----
serial_number    10:39:a7:02:60:b4:b2:22:12:96:b7:b3:0f:7f:c2:79:45:d3:49:fb

Configure a role that maps a name in Vault to a procedure for generating a certificate. When users or machines generate credentials, they are generated against this role:

$ vault write pki/roles/example-dot-com \
                          allowed_domains=my-website.com \
                          allow_subdomains=true \
                          max_ttl=72h
Success! Data written to: pki/roles/example-dot-com

Generate a new credential by writing to the /issue endpoint with the name of the role:

$ vault write pki/issue/example-dot-com \
                        common_name=www.my-website.com
Key                 Value
---                 -----
certificate         -----BEGIN CERTIFICATE-----
MIIDVjCCAj6gAwIBAgIUWQhPLW6R/nk/3x3XReHC1Ze4BWUwDQYJKoZIhvcNAQEL
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
TSuguIiSBt5NN0ou4aY01FbeJJOHZhtpj31XdXOCAKR40lPCmWtEUAbcuEhLlkm+
vmhNYxBqkx33jEIMxk95P4eKIYPyr45/8o7bV1jq7G26aBzj1Mjd0JmU
-----END CERTIFICATE-----
expiration          1574924103
issuing_ca          -----BEGIN CERTIFICATE-----
MIIDPjCCAiagAwIBAgIUEDmnAmC0siISlrezD3/CeUXTSfswDQYJKoZIhvcNAQEL
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
CsFVu+vfMM9XEMYeKHRWAq9onJFyGuwKGhF0/7RbZ3EunTj6Zph+UMucGoL4xfXj
ITltdU1N4JPvihQq+8Omryay
-----END CERTIFICATE-----
private_key         -----BEGIN RSA PRIVATE KEY-----
MIIEpAIBAAKCAQEAuK7V4GuoHSF8pnlr4hApeU7V3zpuQ2rWt3pXgi9TPBCmIuye
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
+o8HetGW5xWvuQ/ObkiSzdQ8nxMyiQj/whe4riYriOw1fYwPrjZfxTm1jsyEmbbm
gYewhfHP3hOgTCVu3SjhvOXS3pnW7hUP4wtvpLLdRumEUM/fK7pwNg==
-----END RSA PRIVATE KEY-----
private_key_type    rsa
serial_number       59:008:4f:2d:6e:91:fe:79:3f:df:1d:d7:45:e1:c2:d5:97:b8:05:65

For more details visit the official Vault documentation.