A Kubernetes liveness and readiness probe is how the kubelet determines health of a pod. This is often times as simple as checking the ability to reach the main service port over TCP or HTTP.
But if you are using Spring Boot and have enabled the Actuator dependency, you have the ability to create even deeper health checks on the service that can be surfaced at ‘/actuator/health’. In turn, this HTTP endpoint can be set as the Kubernetes liveness probe for a deeper determination of pod health.
Spring Boot Docker image prerequisite
As a prerequisite, you will need a Docker image that runs a Spring Boot web application. If you do not have one, you can use my article on building a Docker image from a Spring Boot jar.
The abbreviated instructions are that you need to install:
- OpenJDK 11+
- Docker CE
Then build and push a Spring Boot web app with custom HealthCheck as a Docker image.
git clone https://github.com/fabianlee/spring-boot-with-docker-push.git cd spring-boot-with-docker-push # run app locally with embedded Tomcat server ./gradlew bootJar bootRun # create local Docker image, run container locally ./gradlew docker dockerRun # push Docker image to hub.docker.com sudo docker login -u <userid> -p <password> ./gradlew dockerPush
Spring Boot custom HealthIndicator
If you are using my spring-boot-with-docker-push project, then you already have a custom UserHealthIndicator.java that changes status to DOWN when the user count = 0 and returns a 503 HTTP status code.
If you would like to create a more advanced HealthIndicator, go through my article on implementing a custom HealthIndicator using Spring Boot Actuator.
Kubernetes deployment manifest
To take advantage of your custom health check in Kubernetes, you need to have the liveness probe point at the Actuator health check in the deployment manifest. Per my example in deployment-spring-micro-with-actuator.yaml
apiVersion: apps/v1
kind: Deployment
...
spec:
...
template:
spec:
containers:
- ...
env:
# check for lifetime liveness, restarts if not status 200
livenessProbe:
httpGet:
path: /actuator/health
port: 8081
initialDelaySeconds: 20
periodSeconds: 10
# check for initial readiness, restarts if status not 200
readinessProbe:
httpGet:
path: /actuator/health
port: 8081
initialDelaySeconds: 20
periodSeconds: 10
The reason that we are using port 8081 is because in application.properties, I explicitly set management.server.port to the value below, which puts the Actuator metrics on their own port (the main REST service is on the default port 8080).
management.server.port=8081
This independent port is not required, ‘/actuator’ could just as well have been on the main 8080 port. However, I chose this independent port because I believe it makes it easier to avoid publicly exposing it from the Ingress.
For example, if I have a NGINX Ingress serve up all prefixes under “/” for this service, then actuator and some of its sensitive management components would be exposed also.
Probe evaluation
The kubelet will now query the ‘:8081/actuator/health’ endpoint for a successful HTTP response code of 200.
As per the custom UserHealthIndicator.java if the user count = 0, the health status will change to DOWN and a 503 HTTP response code will be sent.
if (userCount>0)
return new Health.Builder().up().withDetail("usercount", userCount).build();
else
return new Health.Builder().down().withDetail("usercount", userCount).build();
This will cause Kubernetes to see the pod as unhealthy, and the pod will be recreated with the original non-empty User list. The User count will be greater than 0 after the restart, and therefore the pod will become healthy again.
REFERENCES
kubernetes.io, liveness and readiness probe
cloud.google.com, K8S best practices on liveness and readiness probes
Muhammad Ahsan, difference between liveness and readiness probes