Debug Nodes in an AKS Cluster

Tobias Moe Thorstensen -

So you have created and configured an AKS cluster and life is good. You think. As time goes by, you keep adding features to your cluster and eventually something doesnt work as expected. Sounds familiar?

I my particular case I was trying to mount an Azure File Share to my AKS cluster with the NFS protocol, but without any success. I was getting a pretty useful error message when running kubectl describe on the problematic pod:

Output: mount.nfs: access denied by server while mounting storageaccountname.file.core.windows.net:/storageaccountname/fileshareName

Microsoft troubleshooting guide pointed me in the direction that the node probably was missing nfs-common.

To install packages to the underlying Ubuntu Virtual Machine Scale Set I needed SSH access to the node. Pretty straight forward you think? Think again.

This is where this neat little trick comes in handy:

  1. List all your nodes by running kubectl get nodes
  2. Find the node you want to SSH into
  3. Run kubectl debug node/node-name -it --image=mcr.microsoft.com/aks/fundamental/base-ubuntu:v0.0.11
  4. Enjoy root access.

Happy debugging!