How To Install Vault Secrets Operator on Kubernetes

Introduction

Hashicorp's vault is a solution used to manage, store and secure access to your application's secrets. Secrets are sensitive data that you want controlled access to such as passwords, tokens, api keys, certificates etc. Applications running in a Kubernetes cluster can access secrets from vault in various ways, one of which is the Kubernetes Vault Secrets Operator.

A Kubernetes operator is a software extension of kubernetes that provides custom resources for managing applications. The operator installs custom crds in the cluster, enabling developers to interact with the application the same way they would with a native Kubernetes resource. For example, the RabbitMQ Cluster Operator when installed provides custom resources like:

• RabbitmqCluster for creating the cluster
• User for creating users in the cluster

The custom resources of the vault secrets operator manages the syncing of the secrets in a vault namespace with the native Kubernetes secrets resource, making them accessible to workloads in the cluster.

This tutorial will guide you through setting up a dev vault instance and a local Kubernetes cluster, for testing with. You will learn how to install and configure the vault secrets operator on the Kubernetes cluster and finally write a secret in vault to the destination Kubernetes Secret resource.

Prerequisites

To complete this guide, you will need a macos or linux machine with internet and an ngrok tunnel installed.

Step 1 - Setup Vault

The Vault secrets operator doesn't setup up a vault instance. In this tutorial we'll setup the vault instance outside the Kubernetes cluster; directly on the local machine using the official docker image.

docker run --cap-add=IPC_LOCK -e VAULT_LOG_LEVEL=debug -e 'VAULT_DEV_ROOT_TOKEN_ID=myroottoken' -e VAULT_LOCAL_CONFIG='{"storage":{"file":{"path":"/vault/file"}},"ui":true}' -p 8200:8200 hashicorp/vault server -dev

This command runs an in-memory vault server in development mode. The UI can be accessed on http://localhost:8200. Sign in using the root token specified in the command. In this example, it is myroottoken.

By default, a secret engine named secret already exists. The secret engine is of type kv version 2. Create a secret in this engine and provide some secret data. Let's name it intel-service-api.

{
  "DATABASE_HOST": "inteldb.io",
  "DATABASE_PASSWORD": "secretpassword",
  "DATABASE_PORT": "5432",
  "DATABASE_USER": "intel-service-reader"
}

Step 2 - Setup Kubernetes

If you have a Kubernetes cluster already, you can skip this section. If not, setup a local cluster on your machine with k3s.

On macos, install k3d, which is a wrapper for k3s.

brew install k3d

Create a new cluster, give it a name

k3d cluster create intel-demo

Check the nodes

kubectl get nodes

Step - 3 Install Vault Secrets Operator

Install the Vault secrets operator on the intel-demo cluster using the official helm chart.

Add the official helm chart repository

helm repo add hashicorp https://helm.releases.hashicorp.com

Next, install the operator. I'm using the infrastructure-tools namespace and creating it if it doesn't exist. I'm also setting the default connection parameters of the vault instance, through the helm chart values. Because the vault instance is running outside Kubernetes, we will expose it using an ngrok tunnel. Run ngrok http 8200 and provide the tunnel url as the defaultVaultConnection.address value.

helm install --version 0.10.0 --create-namespace --namespace infrastructure-tools vault-secrets-operator hashicorp/vault-secrets-operator \
  --set "defaultVaultConnection.enabled=true" \
  --set "defaultVaultConnection.address=https://<tunnel_id>.ngrok-free.app" \
  --set "defaultVaultConnection.skipTLSVerify=true"

The output on the terminal should have a status of deployed, and the status of the pod kubectl get pods -n infrastructure-tools should be Running.

If the pod remains in a pending state, describe the pod to see the reason. It's likely the k3d node(s) having an untolerated taint e.g {node.kubernetes.io/disk-pressure: }. For such cases, provide a toleration for the taint on the Vault Secrets Operator deployment. E.g

helm upgrade --install --namespace infrastructure-tools vault-secrets-operator hashicorp/vault-secrets-operator \
  --set "tolerations[0].key=node.kubernetes.io/disk-pressure" \
  --set "tolerations[0].operator=Exists" \
  --set "tolerations[0].effect=NoSchedule"

The operator is now installed and the command kubectl get crds | grep vault returns the custom resources provided by the Vault secrets operator. One of them is the vaultconnections (vaultconnections.secrets.hashicorp.com) resource. This resource handles connections from the Kubernetes cluster to the Vault instance. Running kubectl describe vaultconnections default -n infrastructure-tools describes the default connection we provided earlier when the operator was deployed.

Step 4 - Connect Vault Secrets Operator on Kubernetes to Vault Instance

Enable Kubernetes auth method

Vault permits access to it's secrets through a number of authentication methods and one of them is Kubernetes. Visit http://localhost:8200/ui/vault/access and enable the Kubernetes authentication method by configuring with the following values.

path: kube-auth-mount
kubernetes_host: http://host.docker.internal:8001
token_reviewer_jwt: <see instructions below>
ca_certificate: disabled
token_reviewer_jwt: <see instructions below>

Kubernetes host

To get the Kubernetes host, we'll expose the Kubernetes cluster through a local proxy server by running the command.

kubectl proxy --address='0.0.0.0' --port=8001 --accept-hosts='.*'

This allows the Kubernetes api to be reachable from the machine on http://localhost:8001. however, since the vault instance is running in a docker container on the same machine, vault would reach the proxy at http://host.docker.internal:8001.

Token reviewer JWT

The token reviewer jwt is a Kubernetes token used by vault to validate other tokens from the Kubernetes service accounts provided by the VaultAuth resource during authentication. In Kubernetes, tokens are created against a service account. For the reviewer token creation, we will reuse the same service account that the Vault Secrets Operator is deployed in.

Describing the operator pod reveals the service account used by the operator.

kubectl get pod <pod name> -n infrastructure-tools -o jsonpath='{.spec.serviceAccountName}'

In this example, it is named vault-secrets-operator-controller-manager. The following command creates the token.

kubectl create token vault-secrets-operator-controller-manager -n infrastructure-tools

Create Policy

Visit http://localhost:8200/ui/vault/policies/acl and create an ACL policy named intel-policy to allow reading the intel-service-api secret in the secret engine mount.

path "secret/data/intel-service-api" {
   capabilities = ["read", "list"]
}

Create Role

Visit http://localhost:8200/ui/vault/access/kube-auth-mount/item/role/create

name: secret-reader-service
audience: vault
bound_service_account: intel-service-vsa
bound_namespace: intelligence
policy: intel-policy

The bound namespace is the namespace where the secret is to be created.

Synchronize secret

The Kubernetes auth method is now enabled on the vault instance. Let's synchronize the secret we have in vault with a native Kubernetes secret resource in the intelligence namespace.

The command kubectl create ns intelligence creates the intelligence namespace.

First we'll create the VaultAuth custom resource. It is used to authenticate with the Vault instance. In the VaultAuth spec, we'll provide:

• the auth method: which is kubernetes
• the mount: the name of the Kubernetes auth method that was enabled earlier
• the role we created earlier
• a Kubernetes service account
• the Kubernetes namespace to operate in

Apply the manifest file below on the cluster to create the service account and the VaultAuth resource.

kubectl apply vault-auth.yaml

# vault-auth.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: intelligence
  name: intel-service-vsa
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  name: intel-service-vault-auth
  namespace: intelligence
spec:
  method: kubernetes
  mount: kube-auth-mount
  kubernetes:
    role: secret-reader-service
    serviceAccount: intel-service-vsa
    audiences:
      - vault

To sync the secret in Vault to a Kubernetes native secret, the VaultStaticSecret resource is applied on the cluster.

kubectl apply vault-static-secret.yaml

# vault-static-secret.yaml

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: intel-service
  namespace: intelligence
spec:
  # secret engine type
  type: kv-v2

  # name of the secret engine
  mount: secret

  # path of the secret
  path: intel-service-api

  # destination of the native kubernetes secret
  destination:
    name: intel-service
    create: true

  # static secret refresh interval
  refreshAfter: 60s

  # Name of the CRD to authenticate to Vault
  vaultAuthRef: intel-service-vault-auth

Test

Running kubectl describe vaultstaticsecret intel-service -n intelligence should display the config of the vault static secret and the events sections should report that the secret was successfully synced.

Normal SecretSynced 4s VaultStaticSecret Secret synced

To view the plain value of the native kubernetes secret, run:

kubectl get secret intel-service -n intelligence -o jsonpath='{.data.DATABASE_USER}' | base64 --decode

Conclusion

In this tutorial you have learned how to setup a local kubernetes cluster, setup a vault instance and install vault secrets operator on a Kubernetes cluster.