minikube: installing minikube on Mac with secure TLS ingress


Categories: Containers, Mac

minikube makes it easy to spin up a local Kubernetes cluster on macOS, and adding an Ingress is convenient with its built-in Addons.

In this article, I want to take it one step further and show how to expose the Ingress via TLS (secure https) using a custom key/certificate chain.

Prerequisites

  • MacOS
  • Brew package manager
  • QEMU virtual machine emulator/virtualizer
  • ‘step‘ utility for generating custom CA and TLS certificate

Brew

Install Brew as described on its documentation page.  We will use the brew package manager to install the other components needed.

# install brew
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

# add brew environment variables to current shell
eval "$(/opt/homebrew/bin/brew shellenv)"

To persist the proper brew shell settings, add the following values to ~/.zprofile

# brew variables and PATH
eval "$(/opt/homebrew/bin/brew shellenv)"
command -v brew >/dev/null 2>&1 || export PATH=/opt/homebrew/bin:$PATH

QEMU

Install QEMU the open-source virtual machine emulator and virtualizer using brew.

brew install qemu

Then enable the socket_vmnet library for QEMU, which enables the minikube “service” command.

brew install socket_vmnet
brew tap homebrew/services
HOMEBREW=$(which brew) && sudo ${HOMEBREW} services start socket_vmnet

Certificate creation utility

Install the step utility, which makes it convenient to generate certificates.

brew install step

Install minikube

brew install minikube

Start minikube, enable Ingress

# start minikube, override default docker engine and use qemu instead
minikube start --driver=qemu --network socket_vmnet

# validate access to cluster
$ which kubectl
/opt/homebrew/bin/kubectl

$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
minikube Ready control-plane 6m42s v1.28.3

# enable ingress-nginx using Addon
minikube addons enable ingress

# wait for deployment readiness
kubectl rollout status deployment ingress-nginx-controller -n ingress-nginx --timeout=90s

Test Ingress with non-secure http

Before we move forward with secure Ingress over TLS, let’s prove out simple http. We will use similar instructions as described in these tutorials at k8-sdocs and w3cubdocs, however we do need to use an image with an ARM64 target to support Apple Silicon.

Simple example deployed at NodePort

# simple multi-arch deployment that supports ARM64 and AMD64
kubectl create deployment web --image=ghcr.io/fabianlee/google-hello-app-multiarch:1.0.0 --port=8080
kubectl expose deployment web --type=NodePort --port=8080

# wait for full readiness
kubectl rollout status deployment web -n default --timeout=90s

# get NodePort info
kubectl get service web
web_url=$(minikube service web --url)
echo "about to insecurely pull from deployment at NodePort: $web_url"

# use http to pull content from service at NodePort
$ curl $web_url
Hello, world!
Version: 1.0.0
Hostname: web-5d76dc856d-6xhh6

Exposed via Ingress

# set variable used for domain name
domain=hello-world.info

# create Ingress using host 'hello-world.info'
wget https://k8s.io/examples/service/networking/example-ingress.yaml
kubectl apply -f example-ingress.yaml

# wait for ingress to get assigned IP address (will match 'minikube ip')
kubectl get ingress
sleep 60
lb_ip=$(kubectl get ingress -o=jsonpath="{.items[].status.loadBalancer.ingress[0].ip}")
echo "loadbalancer IP for ingress: $lb_ip"

# adding loadbalancer IP to local hosts file
# this way we do not need minikube tunnel
echo $lb_ip $domain | sudo tee -a /etc/hosts

# prove that service is available at Ingress
$ curl http://$domain
Hello, world!
Version: 1.0.0
Hostname: web-5d76dc856d-6xhh6

Create CA cert and TLS certificate

Using the step utility, we can create a custom root CA, intermediate, and leaf certificate for secure TLS at the URL “https://hello-world.info”.

# set variable
domain=hello-world.info

# create root CA
step certificate create --no-password --insecure --profile root-ca "Example Root CA" root_ca.crt root_ca.key

# intermediate cert
step certificate create "Example Intermediate CA 1" intermediate_ca.crt intermediate_ca.key --profile intermediate-ca --ca ./root_ca.crt --ca-key ./root_ca.key --no-password --insecure

# leaf cert
step certificate create $domain $domain.crt $domain.key --profile leaf --not-after=8760h --ca ./intermediate_ca.crt --ca-key ./intermediate_ca.key --bundle --no-password --insecure

# show results
step certificate inspect $domain.crt --short

Load TLS cert and key into cluster

# load key+cert into secret in kube-system namespace
kubectl -n kube-system create secret tls ingress-tls-cert --key $domain.key --cert $domain.crt

# do direct patch to set default TLS certificate (had problem setting with 'minikube addons configure ingress')
kubectl patch deployment ingress-nginx-controller -n ingress-nginx --type "json" --patch '[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--default-ssl-certificate=kube-system/ingress-tls-cert"}]'

# wait for readiness
kubectl rollout status deployment ingress-nginx-controller -n ingress-nginx --timeout=90s

# ensure that secret was loaded by looking at log
$ kubectl logs deployment/ingress-nginx-controller -n ingress-nginx | grep ingress-tls-cert
...
backend_ssl.go:67] "Adding secret to local store" name="kube-system/ingress-tls-cert"

Validate ingress with secure TLS

With the TLS key+cert now loaded, we should be able to pull from the service using secure https.

# pull using TLS (trusted CA cert required)
curl --cacert root_ca.crt https://$domain

# show leaf certificate
step certificate inspect --roots=root_ca.crt --short https://$domain

 

REFERENCES

minikube, introduction

minikube, custom TLS cert with ingress 

devopscube.com, minikube on Mac

minikube.sigs.k8s.io, explains QEMU networking modes and reason for socket_vmnet

github.com, google hello-app but with multiple target architectures

stackoverflow, patch command to set ingress-nginx default certificate

k8s-docs, minikube with ingress nginx

kubernetes.io, minikube and Ingress

github container-hello-app, source code for image gcr.io/google-samples/hello-app

kubernetes.github.io, TLS and default cert

NOTES

If you have a problem where the ingress components never show up, try starting fresh and enabling the ‘ingress-dns’ addon first.

minikube delete
minikube start
minikube addons enable ingress-dns
minikube addons enable ingress

If you have another virtual machine manager (like KVM), you can start minikube like:

minikube start --driver=kvm2

The GoogleCloudPlatform/kubernetes-engine-samples is only for architecture amd64, I modified for multiple target archs

# cannot use on arch64 (apple silicon)
kubectl create deployment web --image=gcr.io/google-samples/hello-app:1.0 --port=8080

# but I have created a multi-arch distribution
kubectl create deployment web --image=ghcr.io/fabianlee/google-hello-app-multiarch:1.0.0 --port=8080

upgrade minikube using brew

minikube version
brew update && brew outdated
minikube delete
brew upgrade qemu socket_vmnet kubernetes-cli
brew upgrade minikube
minikube version
minikube start
minikube logs | grep -i "container runtime version"
minikube logs | grep -i "creating cni manager"