Create a secret provider class to serve secrets from Key Vault to your application

You now have everything in place to start using Key Vault secrets in your application code. You will need to create a secret provider class and update the YAML definition for the config microservice. You can use the below guidance to do so.

Make sure the secrets you created in Key Vault show up as environment variables called SPRING_DATASOURCE_USERNAME and SPRING_DATASOURCE_PASSWORD in your customers, vets and visits pods.

Step by step guidance

As a first step you will create a SecretProviderClass.

ADTENANT=$(az account show --query tenantId --output tsv)
   
cat <<EOF | kubectl apply -n spring-petclinic -f -
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-kvname-user-msi
spec:
  provider: azure
  secretObjects:
  - secretName: gitpatsecret
    type: Opaque
    data: 
    - objectName: gitpat
      key: gitpat
  parameters:
    usePodIdentity: "false"
    useVMManagedIdentity: "false" 
    clientID: $USER_ASSIGNED_CLIENT_ID 
    keyvaultName: $KEYVAULT_NAME
    cloudName: "" 
    objects: |
      array: 
        - |
          objectName: GIT-PAT
          objectType: secret   
          objectAlias: gitpat          
          objectVersion: ""  
    tenantId: $ADTENANT
EOF

This SecretProviderClass connects to your Key Vault and picks up the GIT-PAT Key Vault secrets This will get mapped to a secret object in your AKS cluster. The connection to Key Vault is made with the client ID of the user assigned managed identity you created in a previous step.

You can now update your YAML deployment definition to make use of this secret and map it to an environment variable for your pod. Navigate to the kubernetes directory and update the spring-petclinic-config-server.yml with 1 additional environment variable that maps the gitpatsecret secret created by the SecretProviderClass. Add the below lines after the APPLICATIONINSIGHTS_CONFIGURATION_CONTENT environment variable and before the imagePullPolicy.
```
        - name: GIT_PAT
          valueFrom:
            secretKeyRef:
              name: gitpatsecret
              key: gitpat
```

Also add a azure.workload.identity/use: "true" below line 15 between app: config-server and spec:. The resulting block should look like this:

  template:
    metadata:
      labels:
        app: config-server
        azure.workload.identity/use: "true"

Between the spec and containers config on line 17, add a serviceAccountName: workload-identity-sa and a volumes definition.

      serviceAccountName: workload-identity-sa
      volumes:
      - name: secrets-store01-inline
        csi: 
          driver: secrets-store.csi.k8s.io
          readOnly: true
          volumeAttributes: 
            secretProviderClass: "azure-kvname-user-msi"

This indicates the name of the service account you want to use. The volumes definition creates a volume based on the SecretProviderClass.

And lastly add volumeMounts element after the last environment variable and before the imagePullPolicy. volumeMounts should be at the same level in the YAML file as the imagePullPolicy.

        volumeMounts:
        - name: secrets-store01-inline
          mountPath: "/mnt/secrets-store"
          readOnly: true

Mind the indentation in the Yaml file. Your resulting spring-petclinic-config-server.yml file should look like this:

 apiVersion: apps/v1
 kind: Deployment
 metadata:
   labels:
     app: config-server
   name: config-server
 spec:
   replicas: 1
   selector:
     matchLabels:
       app: config-server
   template:
     metadata:
       labels:
         app: config-server
         azure.workload.identity/use: "true"
     spec:
       serviceAccountName: workload-identity-sa
       volumes:
       - name: secrets-store01-inline
         csi: 
           driver: secrets-store.csi.k8s.io
           readOnly: true
           volumeAttributes: 
             secretProviderClass: "azure-kvname-user-msi"
       containers:
       - image: acrpetclinic38cb12.azurecr.io/spring-petclinic-config-server:3.0.2
         name: config-server
         env:
         - name: "CONFIG_SERVER_URL"
           valueFrom:
             configMapKeyRef:
               name: config-server
               key: CONFIG_SERVER_URL
         - name: "APPLICATIONINSIGHTS_CONNECTION_STRING"
           valueFrom:
             configMapKeyRef:
               name: config-server
               key: APPLICATIONINSIGHTS_CONNECTION_STRING
         - name: "APPLICATIONINSIGHTS_CONFIGURATION_CONTENT"
           value: >-
             {
                 "role": {   
                     "name": "config-server"
                   }
             }
         - name: GIT_PAT
           valueFrom:
             secretKeyRef:
               name: gitpatsecret
               key: gitpat
         volumeMounts:
         - name: secrets-store01-inline
           mountPath: "/mnt/secrets-store"
           readOnly: true
         imagePullPolicy: Always
         livenessProbe:
           failureThreshold: 3
           httpGet:
             path: /actuator/health
             port: 8888
             scheme: HTTP
           initialDelaySeconds: 180
           successThreshold: 1
         readinessProbe:
           failureThreshold: 3
           httpGet:
             path: /actuator/health
             port: 8888
             scheme: HTTP
           initialDelaySeconds: 10
           successThreshold: 1
         ports:
         - containerPort: 8888
           name: http
           protocol: TCP
         - containerPort: 9779
           name: prometheus
           protocol: TCP
         - containerPort: 8778
           name: jolokia
           protocol: TCP
         securityContext:
           privileged: false
       
       
 ---
 apiVersion: v1
 kind: Service
 metadata:
   labels:
     app: config-server
   name: config-server
 spec:
   ports:
   - port: 8888
     protocol: TCP
     targetPort: 8888
   selector:
     app: config-server
   type: LoadBalancer

We are not really using the volumes and volumeMounts elements, since your spring boot app will rely on environment variables. However these elements are needed so the secrets get loaded in your kubernetes cluster. In case you don’t define volumes and volumeMounts you will notive the secretProviderClass being present in the cluster but no additional secrets get created by it. This is because of how the Key Vault CSI driver works.

You will also need to update the application.yml file of the config-server microservice itself, so it does not make use of the hard-coded GitHub PAT token anymore. Navigate to the spring-petclinic-config-server/src/main/resources/application.yml file and update the password of the git repo to use the GIT_PAT environment variable.
```
          password: ${GIT_PAT}
```

Navigate to the root of the application and rebuild the spring-petclinic-config-server.

cd ~/workspaces/java-microservices-aks-lab/src
mvn clean package -DskipTests -rf :spring-petclinic-config-server

You will also have to rebuild the container image for the config-server. Navigate to the acr-staging directory, copy over the compiled jar file and rebuild the container.

   cd staging-acr
   rm spring-petclinic-config-server-$VERSION.jar
   cp ../spring-petclinic-config-server/target/spring-petclinic-config-server-$VERSION.jar spring-petclinic-config-server-$VERSION.jar
   
   docker build -t $MYACR.azurecr.io/spring-petclinic-config-server:$VERSION \
       --build-arg ARTIFACT_NAME=spring-petclinic-config-server-$VERSION.jar \
       --build-arg APP_PORT=8888 \
       --build-arg AI_JAR=ai.jar \
       .

   docker push $MYACR.azurecr.io/spring-petclinic-config-server:$VERSION

Now re-apply the YAML file in your AKS cluster and wait for it to be properly up and running.

   cd ../kubernetes
   kubectl apply -f spring-petclinic-config-server.yml 
   kubectl get pods -w

Once the config-server is properly up and running, escape out of the pod watch statement with Ctrl+Q. Delete the pods of the customers, visits and vets microservices. This will make them restart and reload their configuration

   kubectl delete pod <customers-service-pod-name> 
   kubectl delete pod <vets-service-pod-name> 
   kubectl delete pod <visits-service-pod-name> 

You should see all your pods properly running again and data being shown in the spring petclinic application.
Once you redeployed the microservices, you can check that the secret kubernetes objects got created. In the output you will notice a _gitpatsecret_object.

   kubectl get secrets -n spring-petclinic

In case you see errors or crashloops of your pods, you can use the below statements to diagnose what might be going wrong. A first statement you can try is describe your pod.

   kubectl describe pod <name-of-the-pod> -n spring-petclinic

In the output of the describe statement you should see your environment variables being mapped.

    Environment:
      CONFIG_SERVER_URL:                          <set to the key 'CONFIG_SERVER_URL' of config map 'config-server'>                      Optional: false
      APPLICATIONINSIGHTS_CONNECTION_STRING:      <set to the key 'APPLICATIONINSIGHTS_CONNECTION_STRING' of config map 'config-server'>  Optional: false
      APPLICATIONINSIGHTS_CONFIGURATION_CONTENT:  {
                                                      "role": {   
                                                          "name": "config-server"
                                                        }
                                                  }
      GIT_PAT:                                    <set to the key 'gitpat' in secret 'gitpatsecret'>  Optional: false
      AZURE_CLIENT_ID:                            aeea5848-9f54-498d-a322-040a9cf7d679
      AZURE_TENANT_ID:                            72f988bf-86f1-41af-91ab-2d7cd011db47
      AZURE_FEDERATED_TOKEN_FILE:                 /var/run/secrets/azure/tokens/azure-identity-token
      AZURE_AUTHORITY_HOST:                       https://login.microsoftonline.com/

In the logs of a pod you can see specific errors during startup

   kubectl logs <name-of-the-pod> -n spring-petclinic -f

You can also query the environment variables of your pod.

   kubectl exec -it <name-of-the-pod> -n spring-petclinic -- env 

Here as well you should find back the GIT_PAT environment variable with the correct value.

And lastly, you can connect to the external service IP of the admin server on port 8080 to inspect whether your applications are properly running and what environment variables are loaded.