Istio announced it will support upgrades jumping directly from 1.8 to 1.10, instead of forcing an intermediate upgrade through 1.9.
In this article, I will show you how to do a canary upgrade from a 1.8 operator to 1.10 operator without affecting end user traffic. We will incorporate the new 1.10 concept of revision tags on the namespace instead of using a version.
Prerequisites
I’m going to assume you already have an Istio 1.8.x revisioned control plane.
Download a minimal set of packages and then the github project where I have the Istio operations manifests and scripts.
# get packages sudo apt install curl git unzip -y # download github repo with istio operations file and scripts git clone https://github.com/fabianlee/istio-operator-upgrades.git cd istio-operator-upgrades
Creating revision tag
Newly introduced in Istio 1.10 is the concept of a “revision tag” on a namespace. Instead of adding an “istio.io/rev=1-x-y” label to a namespace indicating the revision of the sidecar to use, you can instead assign a label like “istio.io/rev=stable”.
This allows the namespace label to stay fixed, and instead use istioctl to associate the ‘stable’ label with a specific version.
The usage of this named revision tag is optional, but provides a de-referencing that is beneficial for maintenance and assigns more control to istioctl.
Here is how we would start the installation of 1.10.2, first tagging the older 1.8.1 plane with a “stable” revision tag.
$ export rootdir=$(realpath .) $ export istiover=1.10.2 # install binaries $ curl -L https://istio.io/downloadIstio | ISTIO_VERSION=$istiover sh - # assign 'stable' tag to 1.8.1 $ istio-$istiover/bin/istioctl x revision tag set stable --revision 1-8-1 # tag namespace with new 'stable' label $ kubectl label ns default istio.io/rev=stable --overwrite # view revision tag list, shows association $ istio-$istiover/bin/istioctl x revision tag list TAG REVISION NAMESPACES stable 1-8-1 default # view backing implementation # new entry with objectSelector looking for 'istio.io/rev=stable' $ kubectl get mutatingwebhookconfiguration NAME WEBHOOKS AGE istio-sidecar-injector-1-8-1 1 5m16s istio-revision-tag-stable 2 25s
Installing Istio 1.10.2 revisioned control plane
The proceed with the installation of the newer 1.10.2 control plane.
# create the operator $ istio-$istiover/bin/istioctl operator init --revision 1-10-2 --hub gcr.io/istio-release Using operator Deployment image: gcr.io/istio-release/operator:1.10.2 2021-09-19T22:41:26.458779Z info proto: tag has too few fields: "-" ✔ Istio operator installed ✔ Installation complete # now we have two operators $ kubectl get deployment -n istio-operator NAME READY UP-TO-DATE AVAILABLE AGE istio-operator-1-8-1 1/1 1 1 28m istio-operator-1-10-2 1/1 1 1 61s # create iop object with revisioned control plane $ kubectl create -f istio-operator-1.10.2.yaml istiooperator.install.istio.io/istio-control-plane-1-10-2 created # now we have two iop $ kubectl get iop -A NAMESPACE NAME REVISION STATUS AGE istio-system istio-control-plane-1-8-1 1-8-1 RECONCILING 28m istio-system istio-control-plane-1-10-2 1-10-2 RECONCILING 18s # wait until you see "Ingress gateways installed" ./show-operator-logs.sh 1-10-2 # then wait for all objects to be 'Running' $ watch -n2 kubectl get pods -n istio-system NAME READY STATUS RESTARTS AGE istiod-1-8-1-778cfb9bf6-c4vnq 1/1 Running 0 2m50s istiod-1-10-2-778cfb9bf6-c4vnq 1/1 Running 0 2m50s istio-ingressgateway-6c547694cd-4ppqv 1/1 Running 0 2m46s # show object versions $ ./show-istio-objects.sh
You should see a dual set of control planes now. One for “1-8-x” and another for “1-10-2”.
Restart Deployments with new sidecar
Because we are using a revision tag, instead of directly modifying the namespace label (it is currently tagged with istio.io/rev=stable), we have istioctl reassign ‘stable’ to 1-10-2.
# reassign 'stable' label $ istio-$istiover/bin/istioctl x revision tag set stable --revision 1-10-2 --overwrite # view revision tag list, shows association $ istio-$istiover/bin/istioctl x revision tag list TAG REVISION NAMESPACES stable 1-10-2 default
Do a rolling restart to have your deployments switch over to using the newer sidecar versions.
# rolling all deployments in namespace # which will inject a newer version of the sidecar $ kubectl rollout restart -n default deployments
The new deployments will now have the newer 1-10-2 sidecar.
Delete older control plane
And then remove the older “1-8-x” control plane.
# delete iop, which removes old operator, service,and deployment # but leaves other objects $ timeout 90s kubectl delete -n istio-system iop/istio-control-plane-1-8-1 $ sleep 15 # then remove older control plane objects $ istio-1.8.1/bin/istioctl x uninstall --revision 1-8-1 # only a 1-10-2 control plane is present now $ ./show-istio-objects.sh
REFERENCES
github, istio-operator-upgrades repo
Istio, 1.8 release note changes
Istio, 1.8 operator install and upgrade
Istio, 1.8 direct to 1.10 upgrade
Istio, 1.10 operator install and upgrade
banazicloud.com, istio 1.8 install
NOTES
purge all istio objects from cluster
istio-1.x.y/bin/istioctl x uninstall --purge
force deletion of iop
# try kubectl get -n istio-system iop istio-control-plane-1-7-5 # try harder kubectl delete -n istio-system iop/istio-control-plane-1-7-5 --grace-period=0 --force # last resort kubectl get istiooperator.install.istio.io/istio-control-plane-1-7-5 -n istio-system -o json | jq '.metadata.finalizers = []' | kubectl replace -f -