Istio: Canary upgrade of Operator from Istio 1.8 directly to 1.10

Istio announced it will support upgrades jumping directly from 1.8 to 1.10, instead of forcing an intermediate upgrade through 1.9.

In this article, I will show you how to do a canary upgrade from a 1.8 operator to 1.10 operator without affecting end user traffic.  We will incorporate the new 1.10 concept of revision tags on the namespace instead of using a version.

Prerequisites

I’m going to assume you already have an Istio 1.8.x revisioned control plane.

Download a minimal set of packages and then the github project where I have the Istio operations manifests and scripts.

# get packages
sudo apt install curl git unzip -y

# download github repo with istio operations file and scripts
git clone https://github.com/fabianlee/istio-operator-upgrades.git
cd istio-operator-upgrades

Creating revision tag

Newly introduced in Istio 1.10 is the concept of a “revision tag” on a namespace.  Instead of adding an “istio.io/rev=1-x-y” label to a namespace indicating the revision of the sidecar to use, you can instead assign a label like “istio.io/rev=stable”.

This allows the namespace label to stay fixed, and instead use istioctl to associate the ‘stable’ label with a specific version.

The usage of this named revision tag is optional, but provides a de-referencing that is beneficial for maintenance and assigns more control to istioctl.

Here is how we would start the installation of 1.10.2, first tagging the older 1.8.1 plane with a “stable” revision tag.

$ export rootdir=$(realpath .)
$ export istiover=1.10.2

# install binaries
$ curl -L https://istio.io/downloadIstio | ISTIO_VERSION=$istiover sh -

# assign 'stable' tag to 1.8.1
$ istio-$istiover/bin/istioctl x revision tag set stable --revision 1-8-1

# tag namespace with new 'stable' label
$ kubectl label ns default istio.io/rev=stable --overwrite
# view revision tag list, shows association
$ istio-$istiover/bin/istioctl x revision tag list
TAG REVISION NAMESPACES
stable 1-8-1 default

# view backing implementation
# new entry with objectSelector looking for 'istio.io/rev=stable'
$ kubectl get mutatingwebhookconfiguration

NAME WEBHOOKS AGE
istio-sidecar-injector-1-8-1 1 5m16s
istio-revision-tag-stable 2 25s 

Installing Istio 1.10.2 revisioned control plane

The proceed with the installation of the newer 1.10.2 control plane.

# create the operator
$ istio-$istiover/bin/istioctl operator init --revision 1-10-2 --hub gcr.io/istio-release

Using operator Deployment image: gcr.io/istio-release/operator:1.10.2
2021-09-19T22:41:26.458779Z info proto: tag has too few fields: "-"
✔ Istio operator installed 
✔ Installation complete

# now we have two operators
$ kubectl get deployment -n istio-operator
NAME READY UP-TO-DATE AVAILABLE AGE
istio-operator-1-8-1 1/1 1 1 28m
istio-operator-1-10-2 1/1 1 1 61s

# create iop object with revisioned control plane 
$ kubectl create -f istio-operator-1.10.2.yaml 
istiooperator.install.istio.io/istio-control-plane-1-10-2 created

# now we have two iop
$ kubectl get iop -A
NAMESPACE NAME REVISION STATUS AGE
istio-system istio-control-plane-1-8-1 1-8-1 RECONCILING 28m
istio-system istio-control-plane-1-10-2 1-10-2 RECONCILING 18s

# wait until you see "Ingress gateways installed"
./show-operator-logs.sh 1-10-2

# then wait for all objects to be 'Running'
$ watch -n2 kubectl get pods -n istio-system
NAME READY STATUS RESTARTS AGE
istiod-1-8-1-778cfb9bf6-c4vnq 1/1 Running 0 2m50s
istiod-1-10-2-778cfb9bf6-c4vnq 1/1 Running 0 2m50s
istio-ingressgateway-6c547694cd-4ppqv 1/1 Running 0 2m46s

# show object versions
$ ./show-istio-objects.sh

You should see a dual set of control planes now. One for “1-8-x” and another for “1-10-2”.

Restart Deployments with new sidecar

Because we are using a revision tag, instead of directly modifying the namespace label (it is currently tagged with istio.io/rev=stable), we have istioctl reassign ‘stable’ to 1-10-2.

# reassign 'stable' label
$ istio-$istiover/bin/istioctl x revision tag set stable --revision 1-10-2 --overwrite

# view revision tag list, shows association
$ istio-$istiover/bin/istioctl x revision tag list

TAG REVISION NAMESPACES
stable 1-10-2 default

Do a rolling restart to have your deployments switch over to using the newer sidecar versions.

# rolling all deployments in namespace
# which will inject a newer version of the sidecar
$ kubectl rollout restart -n default deployments

The new deployments will now have the newer 1-10-2 sidecar.

Delete older control plane

And then remove the older “1-8-x” control plane.

# delete iop, which removes old operator, service,and deployment
# but leaves other objects
$ timeout 90s kubectl delete -n istio-system iop/istio-control-plane-1-8-1
$ sleep 15

# then remove older control plane objects
$ istio-1.8.1/bin/istioctl x uninstall --revision 1-8-1

# only a 1-10-2 control plane is present now
$ ./show-istio-objects.sh

 

REFERENCES

github, istio-operator-upgrades repo

Istio, 1.8 release note changes

Istio, 1.8 operator install and upgrade

Istio, 1.8 direct to 1.10 upgrade

Istio, 1.10 operator install and upgrade

istio, 1.10 revision tags

github, istio releases

banazicloud.com, istio 1.8 install

NOTES

purge all istio objects from cluster

istio-1.x.y/bin/istioctl x uninstall --purge

force deletion of iop

# try
kubectl get -n istio-system iop istio-control-plane-1-7-5

# try harder
kubectl delete -n istio-system iop/istio-control-plane-1-7-5 --grace-period=0 --force

# last resort
kubectl get istiooperator.install.istio.io/istio-control-plane-1-7-5 -n istio-system -o json | jq '.metadata.finalizers = []' | kubectl replace -f -