New to KubeVault? Please start here.

Monitor Vault operator with builtin Prometheus

This tutorial will show you how to configure builtin Prometheus scrapper to monitor Vault operator.

Before You Begin

At first, you need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using Minikube.

To keep Prometheus resources isolated, we are going to use a separate namespace to deploy Prometheus server.

$ kubectl create ns monitoring
namespace/monitoring created

Enable Monitoring in Vault operator

Enable Prometheus monitoring using agent while install Vault operator. To know details about how to enable monitoring see here

Here, we are going to enable monitoring for operator metrics.

Using Helm:

$ helm install appscode/vault-operator --name vault-operator --version 0.2.0 --namespace kube-system \
  --set \
  --set monitoring.operator=true \
  --set monitoring.prometheus.namespace=monitoring

Using Script:

$ curl -fsSL  | bash -s -- \ \
  --monitor-operator=true \

This will add necessary annotations to vault-operator service. Prometheus server will scrap metrics using those annotations. Let’s check which annotations are added to the service,

$ kubectl get svc vault-operator -n kube-system -o yaml
apiVersion: v1
kind: Service
  annotations: /metrics "8443" https "true"
  creationTimestamp: "2018-12-26T06:12:51Z"
    app: vault-operator
    chart: vault-operator-0.2.0
    heritage: Tiller
    release: vault-operator
  name: vault-operator
  namespace: kube-system
  resourceVersion: "10030"
  selfLink: /api/v1/namespaces/kube-system/services/vault-operator
  uid: 469d2c8f-08d5-11e9-852c-080027857726
  - name: api
    port: 443
    protocol: TCP
    targetPort: 8443
    app: vault-operator
    release: vault-operator
  sessionAffinity: None
  type: ClusterIP
  loadBalancer: {}

Here, "true" annotation indicates that Prometheus should scrap metrics for this service.

The following three annotations point to api endpoints which provides operator specific metrics. /metrics "8443" https

Now, we are ready to configure our Prometheus server to scrap those metrics.

Deploy Prometheus Server

We have deployed Vault operator in kube-system namespace. Vault exports operator metrics via TLS secured api endpoint. So, Prometheus server need to provide certificate while scrapping metrics from this endpoint. Vault operator has created a secret named vault-operator-apiserver-cert with this certificate in monitoring namespaces as we have specified that we are going to deploy Prometheus in that namespace through --prometheus-namespace or monitoring.prometheus.namespace flag. We have to mount this secret in Prometheus deployment.

Let’s check vault-operator-apiserver-cert secret has been created in monitoring namespace.

$ kubectl get  secrets -n monitoring -l=app=vault-operator
NAME                            TYPE                DATA   AGE
vault-operator-apiserver-cert   2      107m

Create RBAC

If you are using a RBAC enabled cluster, you have to provide necessary RBAC permissions for Prometheus. Following this, let’s create RBAC stuffs for Prometheus by running:

$ kubectl apply -f created
serviceaccount/prometheus created created

YAML for RBAC resources can be found here.

Create ConfigMap

As we are monitoring Vault operator, we should follow this to create a ConfigMap. Bellow the YAML of ConfigMap that we are going to create in this tutorial

apiVersion: v1
kind: ConfigMap
  name: prometheus-config
    name: prometheus-config
  namespace: monitoring
  prometheus.yml: |-
      scrape_interval: 5s
      evaluation_interval: 5s
    - job_name: 'vault-apiservers'
      honor_labels: true
      - role: endpoints
      # Kubernetes apiserver serve metrics on a TLS secure endpoints. so, we have to use "https" scheme
      scheme: https
      # we have to provide certificate to establish tls secure connection
        ca_file: /etc/prometheus/secret/vault-operator-apiserver-cert/tls.crt
        server_name: vault-operator.kube-system.svc
      #  bearer_token_file is required for authorizating prometheus server to Kubernetes apiserver
      bearer_token_file: /var/run/secrets/

      - source_labels: [__meta_kubernetes_service_label_app]
        separator: ;
        regex: vault-operator
        replacement: $1
        action: keep
      - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
        separator: ;
        regex: Node;(.*)
        target_label: node
        replacement: ${1}
        action: replace
      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
        action: keep
        regex: kube-system;vault-operator;api
      - separator: ;
        regex: (.*)
        target_label: endpoint
        replacement: api
        action: replace
      - source_labels: [__meta_kubernetes_service_name]
        separator: ;
        regex: (.*)
        target_label: job
        replacement: ${1}
        action: replace

Look at the tls_config field of vault-apiservers job. We have provided certificate file through ca_file field. This certificate comes from vault-operator-apiserver-cert that we are going to mount in Prometheus deployment. Here, server_name is used to verify hostname. In our case, the certificate is valid for hostname server and vault-operator.kube-system.svc.

In relabel_configs section we added <operator_name>.<namespace>.svc:443 as the value of replacement.

Let’s create the ConfigMap we have shown above,

$ kubectl apply -f
configmap/prometheus-config created

Deploy Prometheus

Now, we are ready to deploy Prometheus server. YAML for the deployment that we are going to create for Prometheus is shown below.

apiVersion: apps/v1
kind: Deployment
    app: prometheus-demo
  name: prometheus
  namespace: monitoring
  replicas: 1
      app: prometheus
        app: prometheus
      - args:
        - --config.file=/etc/prometheus/prometheus.yml
        - --storage.tsdb.path=/prometheus/
        image: prom/prometheus:v2.5.0
        imagePullPolicy: IfNotPresent
        name: prometheus
        - containerPort: 9090
          protocol: TCP
        - mountPath: /etc/prometheus/
          name: prometheus-config
        - mountPath: /prometheus/
          name: prometheus-storage
        - mountPath: /etc/prometheus/secret/vault-operator-apiserver-cert
          name: vault-operator-apiserver-cert
      serviceAccountName: prometheus
      - configMap:
          defaultMode: 420
          name: prometheus-config
        name: prometheus-config
      - emptyDir: {}
        name: prometheus-storage
      - name: vault-operator-apiserver-cert
          defaultMode: 420
          secretName: vault-operator-apiserver-cert
          - path: tls.crt
            key: tls.crt

Notice that, we have mounted vault-operator-apiserver-cert secret as a volume at /etc/prometheus/secret/vault-operator-apiserver-cert directory.

Now, let’s create the deployment,

$ kubectl apply -f
deployment.apps "prometheus" deleted

Verify Monitoring Metrics

Prometheus server is running on port 9090. We are going to use port forwarding to access Prometheus dashboard. Run following commands on a separate terminal,

$ kubectl get pod -n monitoring -l=app=prometheus
NAME                          READY   STATUS    RESTARTS   AGE
prometheus-8568c86d86-vpzx5   1/1     Running   0          102s

$ kubectl port-forward -n monitoring prometheus-8568c86d86-vpzx5  9090
Forwarding from -> 9090
Forwarding from [::1]:9090 -> 9090

Now, we can access the dashboard at localhost:9090. Open http://localhost:9090 in your browser. You should see the configured jobs as target and they are in UP state which means Prometheus is able collect metrics from them.


Cleaning up

To uninstall Prometheus server follow this

To uninstall Vault operator follow this

To cleanup the Kubernetes resources created by this tutorial, run:

$ kubectl delete ns monitoring