The ‘gcloud compute instances move‘ command is convenient for moving VM instances from one region to another, but only works within a narrow scope of OS image types and disks. For example, only older non-UEFI OS images can be moved with this command.
Trying to move even the simplest Ubuntu bionic/focal or Debian bullseye/buster VM from one region to another results in the following error message.
$ gcloud compute instances move instance-1 --zone=us-west4-b --destination-zone=us-east1-b ERROR: (gcloud.compute.instances.move) HTTPError 400: Invalid resource usage: 'Instance is a UEFI-enabled instance and does not support MoveInstance.'.
In this case, you need to perform a manual move of the VM by creating snapshots of the disks, and from that creating new disks in the target zone. Then create a new VM instance in the target zone with the new disks attached.
This is described in the official documentation, and I will present an example below moving a VM instance that has an additional data disk from us-east4-b to us-west4-b.
Setup variables
Define variables required during these instructions.
project_id=$(gcloud config get project) # show OS images available gcloud compute images list --project=$project_id --filter="family~'ubuntu-*'" os_flags="--image-project=ubuntu-os-cloud --image-family=ubuntu-2004-lts" instance_name=vmtest1 machine_type=e2-micro zone=us-east4-b destination_zone=us-west4-b subnet=default
Create Data Disk for VM
In addition to the boot disk, we want to attach a data disk to the VM.
# get list of disk types available gcloud compute disk-types list --filter="zone=$zone" # create 100G data disk that will be ext4 format gcloud compute disks create datadisk1 --type=pd-balanced --zone=$zone --size=100GB # flag that will be used during VM creation data_disk_flag="--disk=device-name=datadisk1,boot=no,mode=rw,name=datadisk1,scope=zonal"
SSH keypair for auth
Create public/private pair for VM authentication.
login_user=ubuntu ssh-keygen -t ed25519 -f gcp-ssh -C $login_user -N "" -q pub_side=$(cat gcp-ssh.pub)
Create VM instance in source zone
First, download my gcp-add-data-disk.sh script that is invoked at VM instance startup and formats/mounts the additional data disk.
wget https://raw.githubusercontent.com/fabianlee/blogcode/master/gcloud/gcp-add-data-disk.sh
Create a VM instance with boot and data disk, using an ssh keypair for authentication, and startup script for additional data disk.
gcloud compute instances create $instance_name $os_flags --project=$project_id --zone=$zone --machine-type=$machine_type --network-interface=network-tier=PREMIUM,subnet=$subnet $data_disk_flag --metadata="os-login=FALSE,ssh-keys=$login_user:$pub_side" --metadata-from-file=startup-script=gcp-add-data-disk.sh
Validate VM
Login to the VM instance and validate data disk mount and zone.
external_ip=$(gcloud compute instances describe $instance_name --zone=$zone --format='value(networkInterfaces[0].accessConfigs[0].natIP)')
# clear any previous fingerprints
ssh-keygen -f ~/.ssh/known_hosts -R $external_ip
# check that ssh is available
nc -vz $external_ip 22
# login to VM instance
ssh $login_user@${external_ip} -i gcp-ssh
# from inside VM
$ df -h /datadisk1
Filesystem Size Used Avail Use% Mounted on
/dev/sdb 98G 61M 98G 1% /datadisk1
# log from gcp-add-data-disk.sh shows VM region
$ cat /datadisk1/hello.txt
2022-05-01 14:21:45
projects/949168343060/zones/us-east4-b
# exit vm shell
exit
Create snapshots of disks
In order to create disks in the destination zone, we have to create snapshots of the disks.
# get name of boot and data disk boot_disk=$(gcloud compute instances describe $instance_name --zone=$zone --format='value(disks[0].source)' | sed 's#.*/##') data_disk=$(gcloud compute instances describe $instance_name --zone=$zone --format='value(disks[1].source)' | sed 's#.*/##') # disks are set to no-delete, so they persist even if VM is deleted for disk in $boot_disk $data_disk; do gcloud compute instances set-disk-auto-delete $instance_name --zone $zone --disk $disk --no-auto-delete; done # save old metadata gcloud compute instances describe $instance_name --zone $zone > /tmp/$instance_name.metadata.old # take snapshots of disks gcloud compute snapshots list for disk in $boot_disk $data_disk; do gcloud compute disks snapshot $disk --snapshot-names snapshot-$disk --zone $zone; done
To show that only the snapshot content is used to create the destination disks, we are going to place another file on the source disk. This will not (and should not) show up on the disks created in the target zone.
# create file, but it is not in snapshot, so will not be moved
ssh $login_user@${external_ip} -i gcp-ssh 'touch /datadisk1/not-part-of-snapshot.txt; ls -l /datadisk1'
Delete original VM and disks
With the snapshots created, we can now completely remove the original source disks and VM instance.
# delete current vm, does not delete disks gcloud compute instances delete $instance_name --zone=$zone --quiet # delete disks in source zone for disk in $boot_disk $data_disk; do gcloud compute disks delete $disk --zone=$zone --quiet; done
Create disks in destination zone
# create new disks in target zone from snapshots for disk in $boot_disk $data_disk; do gcloud compute disks create $disk --source-snapshot snapshot-$disk --zone=$destination_zone; done # show disks in each zone, should be none in source and both in destination gcloud compute disks list --filter="zone~$zone" gcloud compute disks list --filter="zone~$destination_zone"
Create new VM instance in destination zone
# original was e2-tiny target_machine_type=e2-small # boot disk flag boot_disk_flag="--disk name=$boot_disk,boot=yes,mode=rw" # data_disk_flag does not need to be changed # create new VM instance, attach disks gcloud compute instances create $instance_name $os_flags --project=$project_id --zone=$destination_zone --machine-type=$target_machine_type --network-interface=network-tier=PREMIUM,subnet=$subnet $boot_disk_flag $data_disk_flag --metadata="os-login=FALSE,ssh-keys=$login_user:$pub_side" --metadata-from-file=startup-script=gcp-add-data-disk.sh
Validate new VM instance
Now let’s validate that this new VM instance has the expected data disk content.
external_ip=$(gcloud compute instances describe $instance_name --zone=$destination_zone --format='value(networkInterfaces[0].accessConfigs[0].natIP)')
# remove any older fingerprint
ssh-keygen -f ~/.ssh/known_hosts -R $external_ip
# login
ssh $login_user@${external_ip} -i gcp-ssh
# from inside new VM
$ df -h /datadisk1
Filesystem Size Used Avail Use% Mounted on
/dev/sdb 98G 61M 98G 1% /datadisk1
# notice that zone has now changed in last line
$ cat /datadisk1/hello.txt
2022-05-01 11:25:44
projects/949168343060/zones/us-east4-b
2022-05-01 13:40:03
projects/949168343060/zones/us-west4-b
# no 'not-part-of-snapshot.txt' file because it was not part of snapshot
$ ls /datadisk1
hello.txt lost+found
# exit shell
$ exit
You can compare the original metadata against the newer.
# save new metadata gcloud compute instances describe $instance_name --zone $destination_zone > /tmp/$instance_name.metadata.new # compare diff /tmp/$instance_name.metadata.old /tmp/$instance_name.metadata.new
Destroy new VM and disks
# mark disks with auto-delete, so they get deleted with VM for disk in $boot_disk $data_disk; do gcloud compute instances set-disk-auto-delete $instance_name --zone $destination_zone --disk $disk; done # delete snapshots for disk in $boot_disk $data_disk; do gcloud compute snapshots delete snapshot-$disk --quiet; done # delete VM instance along with disks gcloud compute instances delete $instance_name --zone=$destination_zone --quiet # check for any orphaned disks or snapshots gcloud compute disks list gcloud compute snapshots list
REFERENCES
google ref, moving VM to a different region
stackoverflow, moving VM to a different region using snapshots
google ref, gcloud compute instances move
google ref, gcloud compute instances create
uly.me, using ‘gcloud compute instances move’ to move regions
google ref, adding startup script
artark.ca, creating gcp persistent disk, format, mount
devopscube.com, mount extra data disk on GCP
hackingthe.cloud, pulling up metadata on GCP VM instance
stackoverflow, example showing snapshot and then disk creation in different zone
NOTES
check for UEFI enabled OS image
gcloud compute instances describe $instance_name --zone=$zone | grep -i uefi - type: UEFI_COMPATIBLE
check metadata of VM instance
ssh $login_user@${external_ip} -i gcp-ssh 'curl -s http://metadata.google.internal/computeMetadata/v1/instance/zone -H "Metadata-Flavor: Google"; echo ""'
ssh $login_user@${external_ip} -i gcp-ssh 'curl -s http://metadata.google.internal/computeMetadata/v1/instance/machine-type -H "Metadata-Flavor: Google"; echo ""'