Kubernetes: using kubectl to wait for condition of pods, deployments, services

Instead of deploying a pod or service and periodically checking its status for readiness, or having your automation scripts wait for a certain number of seconds before moving to the next operation, it is much cleaner to use ‘kubectl wait’ to sense completion.

Wait for pod

Here is how you would wait for READY status on a newly deployed nginx pod.

# deploy test pod
kubectl run mynginx -n default --image=nginx:latest --restart=Never

# manually list items[].status.conditions[]
kubectl get pod -n default -l run=mynginx -o jsonpath="{.items[*].status.conditions}" | jq

# use 'wait' to check for Ready status in .status.conditions[]
kubectl wait pods -n default -l run=mynginx --for condition=Ready --timeout=90s

Wait for deployment

Another example is checking for a deployment to be fully available.  In this case we have a deployment named golang-hello-world-web.

# test deployment
kubectl apply -f https://raw.githubusercontent.com/fabianlee/k3s-cluster-kvm/main/roles/golang-hello-world-web/templates/golang-hello-world-web.yaml.j2

# restart, then wait using 'rollout status'
kubectl rollout restart deployment golang-hello-world-web -n default
kubectl rollout status deployment golang-hello-world-web -n default --timeout=90s

Wait for service

Services are trickier because ‘wait’ looks for a conditions list, which is not available.  Instead you have .status as shown below.

$ kubectl get service -n ingress-nginx ingress-nginx-controller -o jsonpath="{.status}" | jq

{
  "loadBalancer": {
    "ingress": [
      {
        "ip": "192.168.2.143"
      }
    ]
  }
}

It looks like this will not be addressed until v1.23, so until that time, you can check for a non-empty value like below.

while [[ -z $(kubectl get service -n ingress-nginx ingress-nginx-controller -o jsonpath="{.status.loadBalancer.ingress}" 2>/dev/null) ]]; do
  echo "still waiting for ingress-nginx/ingress-nginx-controller to get ingress"
  sleep 1
done
echo "ingress-nginx/ingress-nginx-controller now has ingress"

 

REFERENCES

kubectl wait

stackovflow, waiting for service assigned to LB

stackoverflow, how to wait for service to get external ip

NOTES

status conditions for pod

# list all status conditions[]
kubectl get pod -n default -l run=mynginx -o jsonpath="{.items[*].status.conditions}" | jq

# get just the 'Ready' condition
kubectl get pod -n default -l run=mynginx -o jsonpath="{range .items[*].status.conditions[?(.type == 'Ready')]}{.type} is {.status}{end}"

status conditions for deployment

# list all conditions[]
kubectl get deployment -n default golang-hello-world-web2 -o jsonpath="{.status.conditions}" | jq

# use range to show all conditions and extract fields
kubectl get deployment -n default golang-hello-world-web2 -o jsonpath="{range .status.conditions[*]}{.type} is {.status}{'\n'}{end}"

# get just the 'Available' condition
kubectl get deployment -n default golang-hello-world-web2 -o jsonpath="{range .status.conditions[?(.type == 'Available')]}{.type} is {.status}{'\n'}{end}"

Manually checking deployment conditions

# manually list items[].status.conditions[]
kubectl get deployment -n default golang-hello-world-web -o jsonpath="{.status.conditions}" | jq

delete objects created in this article

# delete pod
kubectl delete pod mynginx -n default

# delete deployment 
kubectl delete -f https://raw.githubusercontent.com/fabianlee/k3s-cluster-kvm/main/roles/golang-hello-world-web/templates/golang-hello-world-web.yaml.j2

Waiting for deployment like this only works for initial deployment (not rolling)

# this wait for 'Available=True' only works for initial deployment, not rolling
kubectl wait deployment -n default golang-hello-world-web --for condition=Available=True --timeout=90s