Live in harmony with Kubernetes Secrets using Azure Key Vault

Introduction

If you have worked with Kubernetes you probably know that Secrets are not really secrets. They are just base64 encoded key value pairs stored in etcd. Anyone with access to the cluster will have access to these secrets by running a simple command

kubectl get secrets --all-namespaces
kubectl get secrets/database -n my-namespace -t={{.data.password-key}} | base64 -D

In order to safely store these secrets you can enable encryption at rest or configure role based access control(RBAC). This article won't cover these two alternatives as we'll rather look at a helm chart that will injecting secrets stored in Azure Key Vault as environment variables to a pod running Grafana. That being said, this approach can be used for other scenarios where you'll need secrets from Azure Key vault.

Prerequisites

To follow along you'll need to have the following up and running:

Helm installed
An AKS Cluster
An Azure Key Vault instance in the same subscription
A User Assigned Managed Identity with the role Key Vault Secrets Officer to the aforementioned Key Vault. This Managed Identity also needs to be associated with the AKS Cluster.

Configure Kubernetes Cluster

We are using a Helm Chart named akv2k8s to get secrets from Azure Key Vault. Once this chart is installed, two pods will run in our cluster fetching and monitoring changes to specific secrets. This chart will only monitor changes in namespaces with the label

azure-key-vault-env-injection: enabled

To label your namespace in an idempotent way, you can run the following command

kubectl create namespace grafana --dry-run=client -o json|jq '.metadata += {"labels":{"azure-key-vault-env-injection":"enabled"}}' | kubectl apply -f -

Now we'll need to add a reference to our Key Vault.

apiVersion: spv.no/v2beta1
kind: AzureKeyVaultSecret
metadata:
  name: azure-keyvault-secret-inject 
  namespace: grafana
spec:
  vault:
    name: someKeyVaultName # name of key vault
    object:
      name: someKeyVaultName # name of the akv object
      type: secret # akv object type

Apply the changes to our cluster:

kubectl apply -f <name-of-file-above-yml>

Now that we have labeled our namespace and created a reference to a secret in our Azure Key Vault, lets install the Helm Charts!

Installing Helm Charts

Helm is a package managing tool for Kubernets and packages are called Charts. We'll use two Helm Charts to achieve our goal by running Grafana with secrets injected as environment variables to the pod.

Lets start by adding these two charts:

helm repo add grafana https://grafana.github.io/helm-charts
helm repo add spv-charts https://charts.spvapi.no
helm repo update

Lets install these charts

helm upgrade --install akv2k8s spv-charts/akv2k8s --namespace akv2k8s
helm upgrade --install --namespace grafana grafana grafana/grafana

You should now have three pods running. Verify that by running the following commands:

kubectl get pods -n grafana
kubectl get pods -n akv2k8s

Should result in:

grafana-xyz
akv2k8s-controller-xyz
akv2k8s-envinjector-xyz

Configuring Grafana

Lets take a step back and configure Grafana with the values we configured in AzureKeyVaultSecret.

First we'll need to delete Grafana Helm Chart.

helm delete grafana -n grafana

Now we need to customize Grafana by adding the -f option. This option will allow us to pass in a yml file with custom values. We will be using the template found in Grafana's GitHub Repository.

To achieve reading values from Azure Key Vault, we'll be setting the default administrator password for Grafana by passing it in as an environment variable. Find the section for environment variables in the yml file and add the following:

env:
  GF_SECURITY_ADMIN_USER: "admin"
  GF_SECURITY_ADMIN_PASSWORD: "azure-keyvault-secret-inject@azurekeyvault"

GF_SECURITY_ADMIN_PASSWORD will now refer to the name we specified for AzureKeyVaultSecret.

This is unfortunetly not enough as Grafana won't set this environment variable to the administrator password. We'll need to run a command when the pod has started to set a new password.

Under the specificed environment variables add the following:

command:
  - sh
  - -c
  - grafana-cli admin reset-admin-password ${GF_SECURITY_ADMIN_PASSWORD} && /run.sh

This is a workaround, and it seems that wont be fixed in the near future according to this GitHub Issue.

Depending on your configuring, like Ingress, Persist Volum Claims etc, your values.yml file should look something like this:

env:
  GF_SECURITY_ADMIN_USER: "admin"
  GF_SECURITY_ADMIN_PASSWORD: "azure-keyvault-secret-inject@azurekeyvault"

command:
  - sh
  - -c
  - grafana-cli admin reset-admin-password ${GF_SECURITY_ADMIN_PASSWORD} && /run.sh

Lets install Grafanas Helm Chart once more with these values:

helm upgrade --install --namespace grafana grafana grafana/grafana -f values.yml

You should now see an instance of Grafana running in your cluster. Lets inspect the pod by running

kubectl describe pod <name-of-pod> -n grafana

Scroll all the way up to Environment and here you should see your secret being injected:

Now you should be able to access grafana depending on your configuration, by port forwarding or via the ingress, with the password stored in Azure Key Vault.

If you face any issues I would recommend looking at the events section when you describe the pod, or alternatively get the logs by running

kubectl logs <pod-name> -n grafana

Hope you found the post useful, and thank you for reading!