Istio: Canary Operator upgrades between Istio 1.7 minor releases

Istio 1.7 has the ability to do canary upgrades for revisioned control planes and operators.

In this article, I will show you how to do a canary upgrade of the Istio control plane between minor versions without affecting end user traffic.

Prerequisites

Download a minimal set of packages and then the github project where I have the Istio operations manifests and scripts.

# get packages
sudo apt install curl git unzip -y

# download github repo with istio operations file and scripts
git clone https://github.com/fabianlee/istio-operator-upgrades.git
cd istio-operator-upgrades

Installing Istio 1.7.5 revisioned control plane

Install a fully revisioned Istio operator and control plane at revision “1-7-5”

$ export rootdir=$(realpath .)
$ export istiover=1.7.5

# install binaries
$ curl -L https://istio.io/downloadIstio | ISTIO_VERSION=$istiover sh -

# install revisioned operator, first supported in 1.7
$ istio-$istiover/bin/istioctl operator init --revision 1-7-5 --hub gcr.io/istio-release

Using operator Deployment image: gcr.io/istio-release/operator:1.7.5
2021-09-19T22:41:26.458779Z info proto: tag has too few fields: "-"
✔ Istio operator installed 
✔ Installation complete

# now we have two operators
$ kubectl get deployment -n istio-operator
NAME READY UP-TO-DATE AVAILABLE AGE
istio-operator 1/1 1 1 38m
istio-operator-1-7-5 1/1 1 1 61s

# create iop object with revisioned control plane 
$ kubectl create -f istio-operator-1.7.5.yaml 
istiooperator.install.istio.io/istio-control-plane-1-7-5 created

# now we have two iop
$ kubectl get iop -A
NAMESPACE NAME REVISION STATUS AGE
istio-system istio-control-plane HEALTHY 8m21s
istio-system istio-control-plane-1-7-5 1-7-5 RECONCILING 18s

# wait until you see "Ingress gateways installed"
./show-istio-operator-logs.sh 1-7-5

# then wait for all objects to be 'Running'
$ watch -n2 kubectl get pods -n istio-system
NAME READY STATUS RESTARTS AGE
istiod-5ff8b9b9d6-zg4np 1/1 Running 0 8m44s
istiod-1-7-5-778cfb9bf6-c4vnq 1/1 Running 0 2m50s
istio-ingressgateway-6c547694cd-4ppqv 1/1 Running 0 2m46s

# apply namespace label istio.io/rev to default ns
$ kubectl label namespace default istio.io/rev=1-7-5 --overwrite=true

# show object versions
./show-istio-objects.sh

We now have a fully revisioned Istio Operator and control plane.

Canary upgrade  to Istio 1.7.6 revisioned control plane

Now we want to create the new Istio operator and control plane, at revision “1-7-6”.

$ export rootdir=$(realpath .)
$ export istiover=1.7.6

# install binaries
$ curl -L https://istio.io/downloadIstio | ISTIO_VERSION=$istiover sh - 

# install revisioned operator
$ istio-$istiover/bin/istioctl operator init --revision 1-7-6 --hub gcr.io/istio-release

Using operator Deployment image: gcr.io/istio-release/operator:1.7.6
2021-09-19T22:41:26.458779Z info proto: tag has too few fields: "-"
✔ Istio operator installed 
✔ Installation complete

# now we have two operators
$ kubectl get deployment -n istio-operator
NAME READY UP-TO-DATE AVAILABLE AGE
istio-operator-1-7-5 1/1 1 1 28m
istio-operator-1-7-6 1/1 1 1 61s
# create iop object with revisioned control plane 
$ kubectl create -f istio-operator-1.7.6.yaml 
istiooperator.install.istio.io/istio-control-plane-1-7-6 created

# now we have two iop
$ kubectl get iop -A
NAMESPACE NAME REVISION STATUS AGE
istio-system istio-control-plane-1-7-5 1-7-5 RECONCILING 28m
istio-system istio-control-plane-1-7-6 1-7-6 RECONCILING 18s

# wait until you see "Ingress gateways installed"
./show-istio-operator-logs.sh 1-7-6

# then wait for all objects to be 'Running'
$ watch -n2 kubectl get pods -n istio-system
NAME READY STATUS RESTARTS AGE
istiod-1-7-5-778cfb9bf6-c4vnq 1/1 Running 0 38m50s
istio-ingressgateway-6c547694cd-4ppqv 1/1 Running 0 2m46s
istiod-1-7-6-59ff6d5579-x2snt 1/1 Running 0 86s

# apply namespace label istio.io/rev to default ns
$ kubectl label namespace default istio.io/rev=1-7-6 --overwrite=true

# show object versions
./show-istio-objects.sh

You should see a dual set of control planes now.  One for “1-7-5” and another for “1-7-6”.

Restart Deployments with new sidecar

Do a rolling restart to have your deployments switch over to using the newer sidecar versions.

# restart of all deployments in default namespace
# which will inject a newer version of the sidecar
$ kubectl rollout restart -n default deployments

Delete older control plane

And then remove the older “1-7-5” control plane.

# delete iop, which removes old operator, service,and deployment
# but leaves other objects
$ timeout 120s kubectl delete -n istio-system iop/istio-control-plane-1-7-5
$ sleep 15

# then remove older control plane objects
$ istio-1.7.5/bin/istioctl x uninstall --revision 1-7-5

# only a 1-7-6 control plane is present now
$ ./show-istio-objects.sh

Canary upgrade  to Istio 1.7.8 revisioned control plane

Let’s go through one more minor version just to practice the process.  Let’s do another canary upgrade from 1.7.6 to 1.7.8.

$ export rootdir=$(realpath .)
$ export istiover=1.7.8

# install binaries
$ curl -L https://istio.io/downloadIstio | ISTIO_VERSION=$istiover sh - 

# install revisioned operator
$ istio-$istiover/bin/istioctl operator init --revision 1-7-8 --hub gcr.io/istio-release

Using operator Deployment image: gcr.io/istio-release/operator:1.7.6
2021-09-19T22:41:26.458779Z info proto: tag has too few fields: "-"
✔ Istio operator installed 
✔ Installation complete

# now we have two operators
$ kubectl get deployment -n istio-operator
NAME READY UP-TO-DATE AVAILABLE AGE
istio-operator-1-7-6 1/1 1 1 28m
istio-operator-1-7-8 1/1 1 1 61s

# create iop object with revisioned control plane 
$ kubectl create -f istio-operator-1.7.8.yaml 
istiooperator.install.istio.io/istio-control-plane-1-7-8 created

# now we have two iop
$ kubectl get iop -A
NAMESPACE NAME REVISION STATUS AGE
istio-system istio-control-plane-1-7-6 1-7-6 RECONCILING 28m
istio-system istio-control-plane-1-7-8 1-7-8 RECONCILING 18s

# wait until you see "Ingress gateways installed"
./show-istio-operator-logs.sh 1-7-8

# then wait for all objects to be 'Running'
$ watch -n2 kubectl get pods -n istio-system
NAME READY STATUS RESTARTS AGE
istiod-1-7-6-778cfb9bf6-c4vnq 1/1 Running 0 38m50s
istio-ingressgateway-6c547694cd-4ppqv 1/1 Running 0 2m46s
istiod-1-7-8-59ff6d5579-x2snt 1/1 Running 0 86s

# apply namespace label istio.io/rev to default ns
$ kubectl label namespace default istio.io/rev=1-7-8 --overwrite=true

# show object versions
./show-istio-objects.sh

You should see a dual set of control planes now.  One for “1-7-6” and another for “1-7-8”.

Restart Deployments with new sidecar

Do a rolling restart to have your deployments switch over to using the newer sidecar versions.

# example of rolling deployment
# which will inject a newer version of the sidecar
$ kubectl rollout restart -n default deployments

Delete older control plane

And then remove the older “1-7-6” control plane.

# delete iop, which removes old operator, service,and deployment
# but leaves other objects
$ timeout 120s kubectl delete -n istio-system iop/istio-control-plane-1-7-6
$ sleep 15

# then remove older control plane objects
$ istio-1.7.6/bin/istioctl x uninstall --revision 1-7-6

# only a 1-7-8 control plane is present now
$ ./show-istio-objects.sh

 

You have now been through two canary upgrades of the control plane, and end user traffic should not have been affected by any of these operations.

 

Scripted

All the manual steps in this article are wrapped up into scripts in my github repo.  The commands directly below are equivalent to running all the manual steps in this article.

# install revisioned operator
./istio-install-operator.sh 1.7.5

# do canary upgrades
./istio-canary-operator-upgrade.sh 1.7.5 1.7.6
./istio-canary-operator-upgrade.sh 1.7.6 1.7.8

 

 

REFERENCES

github, istio-operator-upgrades repo

Istio, 1.7 release note changes

Istio, 1.7 upgrade

Istio, 1.7 operator install and upgrade

Istio, 1.8 release note changes

github, istio releases

NOTES

force deletion of iop

# try
kubectl get -n istio-system iop istio-control-plane-1-7-5

# try harder
kubectl delete -n istio-system iop/istio-control-plane-1-7-5 --grace-period=0 --force

# last resort
kubectl get istiooperator.install.istio.io/istio-control-plane-1-7-5 -n istio-system -o json | jq '.metadata.finalizers = []' | kubectl replace -f -