How to take thread dumps for a java service running in k8s

Recently I came across this interesting problem, where I wanted to take a thread dump of a java service. Our service was restarting multiple times because of which were were not able to download the thread dump at the time of CPU outage. This blogs covered how we were finally able to get the thread dump.

We used the Persistent Volume concept from the Kubernetes.

Concepts

• PersistentVolume: This is the actual store where we will store the data. Conceptually it is similar to nodes, i.e. we don’t actually interact much with nodes, we have an abstract concept pod built on top of it.

• PersistentVolumeClaim: We are claiming some volume. In pods, we can directly refer to the claim.

Steps to create an link a Persistent Volume

1. Create a Persistent Volume Claim using kubectl apply -f pvc-demo.yaml.

# pvc-demo.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-demo-2
  namespace: default
spec:
  storageClassName: standard
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 75Gi

1. Check that the PVC is created kubectl get pvc --namespace default.

2. Change the deployment config to use this PVC in the volumes section.

 - name: heap-dumps
    persistentVolumeClaim:
    claimName: pvc-demo-2

1. Redeploy the service using kubectl apply -f manager.yaml.

2. Now all the data inside heapdump is preserved.

Hope you find this blog useful when taking a thread dump for a service which is restarting frequently.