Istio 1.7 has the ability to do canary upgrades for revisioned control planes and operators, but if you did your initial installation without the ‘revision’ flag, then you’ll need to apply these settings.
In this article, I will show you how to go from an non-revisioned 1.7.5 Istio operator and control plane to a 1.7.5 fully revisioned operator and control plane objects.
Prerequisites
Download a minimal set of packages and then the github project where I have the Istio operations manifests and scripts.
# get packages sudo apt install curl git unzip -y # download github repo with istio operations file and scripts git clone https://github.com/fabianlee/istio-operator-upgrades.git cd istio-operator-upgrades
Installing Istio 1.7 non-revisioned
Install with a non-revisioned operator and control plane. Avoid setting ‘spec.revision’ in the operator manifest as well, so that none of the control plane objects are revisioned.
$ export rootdir=$(realpath .) $ export istiover=1.7.5 # install binaries $ curl -L https://istio.io/downloadIstio | ISTIO_VERSION=$istiover sh - # install with non-revisioned operator $ istio-$istiover/bin/istioctl operator init --hub gcr.io/istio-release Using operator Deployment image: gcr.io/istio-release/operator:1.7.5 ✔ Istio operator installed ✔ Installation complete # create namespace $ kubectl create ns istio-system namespace/istio-system created # apply operator manifest # purposely avoid using '.spec.revision' # this means our control plane objects will also be non-revisioned $ kubectl create -f istio-operator-1.7.5-no-revision.yaml istiooperator.install.istio.io/istio-control-plane created # wait until you see "Ingress gateways installed" $ ./show-operator-logs.sh default # wait for all objects to be READY $ kubectl get pods -n istio-system NAME READY STATUS RESTARTS AGE istiod-65b57c788-df9pc 1/1 Running 0 41s istio-ingressgateway-84cc5bf55b-qkg8n 1/1 Running 0 36s # set non-revisioned style labels on namespace # used for envoy sidecar auto-injection $ kubectl label namespace istio-operator istio-injection=disabled --overwrite=true $ kubectl label namespace default istio-injection=enabled --overwrite=true $ kubectl get namespace -L istio-injection # because of no revision definition # there are no versions on: operator, iop, istiod, mutatingwebhook ./show-istio-objects.sh
This has now provided us a working control plane, but it is non-revisioned.
Upgrade to 1.7 fully revisioned control plane
We now run the ‘operator init’ command again, but this time with a ‘revision’ flag.
$ export rootdir=$(realpath .) $ export istiover=1.7.5 # install revisioned operator, first supported in 1.7 $ istio-$istiover/bin/istioctl operator init --revision 1-7-5 --hub gcr.io/istio-release Using operator Deployment image: gcr.io/istio-release/operator:1.7.5 2021-09-19T22:41:26.458779Z info proto: tag has too few fields: "-" ✔ Istio operator installed ✔ Installation complete # now we have two operators $ kubectl get deployment -n istio-operator NAME READY UP-TO-DATE AVAILABLE AGE istio-operator 1/1 1 1 38m istio-operator-1-7-5 1/1 1 1 61s # create iop object with revisioned control plane $ kubectl create -f istio-operator-1.7.5.yaml istiooperator.install.istio.io/istio-control-plane-1-7-5 created # now we have two iop $ kubectl get iop -A NAMESPACE NAME REVISION STATUS AGE istio-system istio-control-plane HEALTHY 8m21s istio-system istio-control-plane-1-7-5 1-7-5 RECONCILING 18s # wait until you see "Ingress gateways installed" $ ./show-operator-logs.sh 1-7-5 # then wait for all objects to be 'Running' $ watch -n2 kubectl get pods -n istio-system NAME READY STATUS RESTARTS AGE istiod-5ff8b9b9d6-zg4np 1/1 Running 0 8m44s istiod-1-7-5-778cfb9bf6-c4vnq 1/1 Running 0 2m50s istio-ingressgateway-6c547694cd-4ppqv 1/1 Running 0 2m46s # remove non-revisioned style injection labels kubectl label namespace default istio-injection- kubectl label namespace istio-operator istio-injection- # apply revisioned label to default ns $ kubectl label namespace default istio.io/rev=1-7-5 --overwrite=true $ kubectl get namespace -L istio.io/rev # show object versions ./show-istio-objects.sh
Now you comprehensively have duplicate control plane objects for the default and 1-7-5 revision.
Restart Deployments with new sidecar
Although the sidecar container image is the same version, do a rolling restart to have your deployments switch over to using the newer revision label (instead of istio.io/rev=default).
# restart of all deployments in default namespace # which will inject a newer version of the sidecar $ kubectl rollout restart -n default deployments
Delete older control plane
When satisfied with the results, we can get rid of all the non-revision components.
# delete non-revisioned iop $ timeout 90s kubectl delete -n istio-system iop/istio-control-plane # delete non-revisioned operator $ kubectl delete deployment/istio-operator -n istio-operator $ kubectl delete service/istio-operator -n istio-operator # delete non-revisioned objects $ istio-1.7.5/bin/istioctl x uninstall --filename istio-operator-1.7.5-no-revision.yaml # show object versions # only the revisioned 1-7-5 objects remain $ ./show-istio-objects.sh
Now we have revisions applied to both the operator and full control plane. Future canary upgrades can be done as described in the official documentation.
Scripted
All the manual steps in this article are wrapped up into scripts in my github repo. The command directly below is equivalent to running all the manual steps in this article.
# install 1.7.5 non-revisioned, change to revisioned ./istio-install-nonrev-operator-to-rev.sh 1.7.5
REFERENCES
github, istio-operator-upgrades repo
Istio, 1.7 release note changes
Istio, 1.7 operator install and upgrade
Istio, 1.8 release note changes
NOTES
force deletion of iop
# try kubectl get -n istio-system iop istio-control-plane-1-6-6 # try harder kubectl delete -n istio-system iop/istio-control-plane-1-6-6 --grace-period=0 --force # last resort kubectl get istiooperator.install.istio.io/istio-control-plane -n istio-system -o json | jq '.metadata.finalizers = []' | kubectl replace -f -