Secure Secrets Management in Kubernetes with HashiCorp Vault: A Comprehensive Guide

As cloud-native applications become more prevalent, organizations are increasingly adopting Kubernetes to orchestrate their containerized workloads. One of the key challenges that come with Kubernetes is managing secrets securely. In this blog post, we’ll explore how to handle secrets in Kubernetes using HashiCorp Vault. We'll go through an introduction to HashiCorp Vault, setting it up in a Kubernetes cluster, and demonstrate a practical example of integrating it with your applications.

Why Use HashiCorp Vault?

HashiCorp Vault is a tool for securely accessing secrets. A secret is anything that you want to tightly control access to, such as API encryption keys, passwords, or certificates. HashiCorp Vault provides a unified interface to any secret while providing tight access control and recording a detailed audit log. Key benefits include:

• Centralized Management: Manage all secrets in a single place.
• Access Control: Fine-grained control over who has access to secrets.
• Audit Logs: Detailed logs of access and usage for compliance.
• Dynamic Secrets: Generate secrets on-demand that are time-scoped and automatically expired.

Prerequisites

• A Kubernetes cluster
• kubectl installed and configured
• Helm installed

Step-by-Step Guide to Integrating HashiCorp Vault with Kubernetes

Step 1: Install HashiCorp Vault Using Helm

First, we will install HashiCorp Vault using Helm. Add the HashiCorp Helm repository:

helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update

Then, install Vault:

helm install vault hashicorp/vault --namespace vault --create-namespace

Step 2: Initialize and Unseal Vault

Initialize Vault so that it can start managing secrets:

kubectl exec -it vault-0 -n vault -- vault operator init

This command will output unseal keys and the initial root token. Store these securely. Unseal Vault using the unseal keys:

kubectl exec -it vault-0 -n vault -- vault operator unseal [Unseal Key 1]
kubectl exec -it vault-0 -n vault -- vault operator unseal [Unseal Key 2]
kubectl exec -it vault-0 -n vault -- vault operator unseal [Unseal Key 3]

Step 3: Enable Kubernetes Authentication

Configure Vault to authenticate using Kubernetes Service Accounts:

kubectl exec -it vault-0 -n vault -- vault auth enable kubernetes

Retrieve Kubernetes token and CA certificate details and configure Vault:

VAULT_SA_NAME=$(kubectl get sa vault -o jsonpath="{.secrets[0].name}" -n vault)
SA_JWT_TOKEN=$(kubectl get secret $VAULT_SA_NAME -o jsonpath="{.data.token}" -n vault | base64 --decode)
SA_CA_CRT=$(kubectl get secret $VAULT_SA_NAME -o jsonpath="{.data['ca.crt']}" -n vault | base64 --decode)
VAULT_ADDR=$(kubectl get svc vault -o jsonpath="{.spec.clusterIP}" -n vault)

kubectl exec -it vault-0 -n vault -- vault write auth/kubernetes/config \
  token_reviewer_jwt="$SA_JWT_TOKEN" \
  kubelet_ca_cert="$SA_CA_CRT" \
  kubernetes_host="https://$VAULT_ADDR:443"

Step 4: Define Roles in Vault

Define roles in Vault that map Kubernetes Service Accounts to policies:

kubectl exec -it vault-0 -n vault -- vault write auth/kubernetes/role/my-role \
  bound_service_account_names=vault \
  bound_service_account_namespaces=default \
  policies=default \
  ttl=1h

Step 5: Creating and Storing Secrets

Create and store a secret in Vault:

kubectl exec -it vault-0 -n vault -- vault kv put secret/myapp/config myapp-secret="s3cr3t"

Step 6: Fetching Secrets from an Application

In your application deployment YAML, add annotations to fetch secrets from Vault:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "my-role"
        vault.hashicorp.com/agent-inject-secret-myapp-config: "secret/myapp/config"
    spec:
      serviceAccountName: vault
      containers:
      - name: myapp
        image: myapp:latest
        env:
        - name: MYAPP_SECRET
          valueFrom:
            secretKeyRef:
              name: myapp-secrets
              key: myapp-secret

This setup allows your application to securely fetch secrets from Vault and inject them as environment variables.

Verifying the Setup

To verify that your setup is working correctly, deploy the application and check logs to ensure that the secrets are injected properly:

kubectl get pods
kubectl logs [myapp-pod-name]

Lessons Learned

Here are a few lessons learned during the integration process:

• Security: Always secure your unseal keys and root tokens, as they are critical for Vault security.
• Automation: Automating the initialization and unsealing of Vault can streamline deployments.
• Monitoring: Regularly monitor your Vault setup to ensure there are no unauthorized access attempts.
• Documentation: Keep your team well-informed and trained on using Vault for managing secrets.

Conclusion

HashiCorp Vault provides a robust solution for managing secrets in Kubernetes, enhancing security, and simplifying secret management workflows. By following the steps outlined in this guide, you can integrate Vault into your Kubernetes environment efficiently. If you have any questions or experiences to share, feel free to leave a comment below.