Istio: Upgrading from Istio 1.6 operator without revision to 1.7 fully revisioned control plane

Istio has the ability to do canary upgrades for revisioned control planes, but it was only in 1.7 that the Operator itself got  support for the ‘revision’ flag.

In this article, I will show you how to go from an non-revisioned 1.6.6 Istio operator and control plane to a 1.7 revisioned operator and control plane objects.

Prerequisites

Download a minimal set of packages and then the github project where I have the Istio operations manifests and scripts.

# get packages
sudo apt install curl git unzip -y

# download github repo with istio operations file and scripts
git clone https://github.com/fabianlee/istio-operator-upgrades.git
cd istio-operator-upgrades

Installing Istio 1.6 non-revisioned

Install with a non-revisioned operator (1.6 did not support revisioned operators).  And then avoid setting ‘spec.revision’ in the operator manifest as well, so that none of the control plane objects are revisioned.

$ export rootdir=$(realpath .)
$ export istiover=1.6.6

# install binaries 
$ curl -L https://istio.io/downloadIstio | ISTIO_VERSION=$istiover sh -

# install with non-revisioned operator
$ istio-$istiover/bin/istioctl operator init --hub gcr.io/istio-release
Using operator Deployment image: gcr.io/istio-release/operator:1.6.6
✔ Istio operator installed 
✔ Installation complete

# create namespace
$ kubectl create ns istio-system
namespace/istio-system created

# apply operator manifest 
# purposely avoid using '.spec.revision'
# this means our control plane objects will also be non-revisioned
$ kubectl apply -f istio-operator-1.6.6-no-revision.yaml
istiooperator.install.istio.io/istio-control-plane created

# wait for all objects to be READY
$ kubectl get pods -n istio-system
NAME READY STATUS RESTARTS AGE
istiod-65b57c788-df9pc      1/1 Running 0 41s
istio-ingressgateway-84cc5bf55b-qkg8n 1/1 Running 0 36s

# set non-revisioned style labels on namespace
# used for envoy sidecar auto-injection
$ kubectl label namespace istio-operator istio-injection=disabled --overwrite=true
$ kubectl label namespace default istio-injection=enabled --overwrite=true
$ kubectl get namespace -L istio-injection

# because of no revision definition,
# there are no versions on: operator, iop, istiod, mutatingwebhook
$ ./show-istio-objects.sh

This has now provided us a working control plane, but it is completely non-revisioned.

Prepare for 1.7 revisioned control plane

In order to prepare 1.6 for a 1.7 upgrade, we will make the following changes:

  • Introduce the revisioned style of namespace labels ‘istio.io/rev’, and remove the older labels
  • Set the ‘.spec.revision’ to ‘1-6-6’ so the control plane objects are revisioned (this will not revision the operator)
  • enable the pilot add-on component which will hang the 1.7 upgrade if not included
# remove non-revisioned style injection labels
kubectl label namespace default istio-injection-
kubectl label namespace istio-operator istio-injection-

# add revisioned style namespace label
kubectl label namespace default istio.io/rev=1-6-6 --overwrite=true
kubectl label namespace istio-operator istio.io/rev=1-6-6 --overwrite=true

# apply changes to components in readiness for 1.7
$ kubectl apply -f istio-operator-1.6.6.yaml
istiooperator.install.istio.io/istio-control-plane-1-6-6 created

# show objects
./show-istio-objects.sh

Notice there are now both default as well as revisioned (1-6-6) objects in the control plane now.

Go ahead and delete all the default control plane objects by deleting the non-revisioned iop object, which will cascade a deletion of all its control plane objects.

# delete non-revisioned iop, cascaded deletion
$ timeout 120s kubectl delete -n istio-system iop/istio-control-plane

# the only object that is non-revisioned is 'istio-operator'
$ ./show-istio-objects.sh

Now, you are ready to install a full revisioned 1.7 operator and control plane.

Deploy 1.7 fully revisioned control plane

Starting in 1.7, the ‘–revision’ flag became available for the operator object as well.

$ export rootdir=$(realpath .)
$ export istiover=1.7.5

# install binaries 
$ curl -L https://istio.io/downloadIstio | ISTIO_VERSION=$istiover sh -

# install revisioned operator, first supported in 1.7
$ istio-$istiover/bin/istioctl operator init --revision 1-7-5 --hub gcr.io/istio-release

Using operator Deployment image: gcr.io/istio-release/operator:1.7.5
2021-09-19T22:41:26.458779Z info proto: tag has too few fields: "-"
✔ Istio operator installed 
✔ Installation complete

# now we have two operators
$ kubectl get deployment -n istio-operator
NAME READY UP-TO-DATE AVAILABLE AGE
istio-operator 1/1 1 1 38m
istio-operator-1-7-5 1/1 1 1 61s

# create iop object with revisioned control plane 
$ kubectl create -f istio-operator-1.7.5.yaml
istiooperator.install.istio.io/istio-control-plane-1-7-5 created

# now we have two iop
$ kubectl get iop -A
NAMESPACE NAME REVISION AGE
istio-system istio-control-plane-1-6-6 1-6-6 12m
istio-system istio-control-plane-1-7-5 1-7-5 2m59s

# wait until you see "Ingress gateways installed"
$ ./show-operator-logs.sh 1-7-5

# then wait for all objects to be 'Running'
$ watch -n2 kubectl get pods -n istio-system
NAME READY STATUS RESTARTS AGE
istio-ingressgateway-6b48bb4d46-n98cr 1/1 Running 0 41s
istiod-1-6-6-56c494d958-qp9j6 1/1 Running 0 3m47s
istiod-1-7-5-645d5c6d46-jrzmw 1/1 Running 0 50s

# apply new namespace label to default ns
$ kubectl label namespace default istio.io/rev=1-7-5 --overwrite=true
$ kubectl get namespace -L istio.io/rev

# show object versions
$ ./show-istio-objects.sh

Now you comprehensively have duplicate control plane objects for 1-6-6 and 1-7-5.

Restart Deployments with new sidecar

Do a rolling restart to have your deployments switch over to using the newer sidecar versions.

# restart of all deployments in default namespace 
# which will inject a newer version of the sidecar 
$ kubectl rollout restart -n default deployments

Delete older control plane

When satisfied with the results,  we can get rid of all the 1-6-6 components and default operator.

# delete iop
$ timeout 120s kubectl delete -n istio-system iop/istio-control-plane-1-6-6

# show object versions
# only the default non-revisioned operator remains
$ ./show-istio-objects.sh

# delete old operator
# do not use 'istioctl operator remove', that would remove even 1.7
kubectl delete deployment/istio-operator -n istio-operator
kubectl delete service/istio-operator -n istio-operator

# only revisioned, 1-7-5 control plane objects remain
$ ./show-istio-objects.sh

 

Future canary upgrades can be now be performed as described in the official documentation.

Scripted

All the manual steps in this article are wrapped up into scripts in my github repo.  The commands directly below are equivalent to running all the manual steps in this article.

# install 1.6.6 non-revisioned, change to revisioned
./istio-install-nonrev-operator-to-rev.sh 1.6.6

# do canary upgrade from 1.6.6 to 1.7.5
./istio-canary-operator-upgrade.sh 1.6.6 1.7.5

 

REFERENCES

github, istio-operator-upgrades repo

Istio, 1.6 upgrade

Istio, 1.6 operator install and upgrade

Istio, 1.7 release note changes

Istio, 1.7 upgrade

Istio, 1.7 operator install and upgrade

Istio, 1.8 release note changes

github, istio releases

https://itnext.io/ingress-for-anthos-multi-cluster-ingress-and-global-service-load-balancing-c56c57b97e82

NOTES

list all containers in a cluster

kubectl get pods --all-namespaces -o jsonpath="{.items[*].spec.containers[*].image}" |\
tr -s '[[:space:]]' '\n' |\
sort |\
uniq -c

wait for deployment to be ready [1]

while [[ $(kubectl get pods -l app=hello -o 'jsonpath={..status.conditions[?(@.type=="Ready")].status}') != "True" ]]; do echo "waiting for pod"; sleep 1; done

force deletion of iop

# try
kubectl get -n istio-system iop istio-control-plane-1-6-6

# try harder
kubectl delete -n istio-system iop/istio-control-plane-1-6-6 --grace-period=0 --force

# last resort
kubectl get istiooperator.install.istio.io/istio-control-plane -n istio-system -o json | jq '.metadata.finalizers = []' | kubectl replace -f -