Terraform is a popular tool for provisioning infrastructure on cloud provider such as EC2 and Azure, but there is also a provider written for local KVM libvirt resources.
Using the libvirt provider, we can use standard Terraform constructs to create local VMs, networks, and disks.
Prerequisite KVM
As a prerequisite for this article, you must install KVM and libvirt as described here.
Install Terraform
Find the latest version of the binaries at the Terraform download page, download.
sudo apt-get install jq unzip -y
# explicitly choose version OR pull latest using github api
export TERRA_VERSION=0.12.20
export TERRA_VERSION=$(curl -sL https://api.github.com/repos/hashicorp/terraform/releases/latest | jq -r ".tag_name" | cut -c2-)
# download
wget https://releases.hashicorp.com/terraform/${TERRA_VERSION}/terraform_${TERRA_VERSION}_linux_amd64.zip
# unzip
unzip terraform_${TERRA_VERSION}_linux_amd64.zip
# set permissions and move into path
chmod +x terraform
sudo mv terraform /usr/local/bin/.
# validate
terraform version
Install libvirt provider plugin
Use prebuilt
If you are using Ubunti bionic 18.04, then go to the latest “releases” page for the terraform-provider-libvirt project, and download the latest binary plugin.
Copy the plugin into the “~/.terraform.d” plugins directory
# make plugins directory mkdir -p ~/.terraform.d/plugins && cd $_ # download wget https://github.com/dmacvicar/terraform-provider-libvirt/releases/download/v0.6.1/terraform-provider-libvirt-0.6.1+git.1578064534.db13b678.Ubuntu_18.04.amd64.tar.gz # extract tar xvfz terraform*.tar.gz
Build from source
However, if you are not lucky enough to have a release already built, you can build from source. I have provided a Docker image that can build the plugin for Ubuntu xenial/Debian stretch.
cd ~ # download project git clone https://github.com/fabianlee/terraform-libvirt-provider-builder.git cd terraform-libvirt-provider-builder # build plugin using docker image sudo apt-get install -y make git libvirt-dev make -f Makefile.stretch # check required dynamically loaded libraries ldd terraform-provider-libvirt # check plugin output (should say 'This binary is a plugin') ./terraform-provider-libvirt This binary is a plugin. These are not meant to be executed directly. Please execute the program that consumes these plugins, which will load any plugins automatically # make plugins directory, copy plugin mkdir -p ~/.terraform.d/plugins && cd $_ cp terraform-provider-libvirt ~/.terraform.d/plugins/.
Create simple VM using Terraform
Let’s start with a simple test, I have a github project named terraform-libvirt-ubuntu-examples that contains various examples of terraform usage with KVM.
sudo apt-get install git make git clone https://github.com/fabianlee/terraform-libvirt-ubuntu-examples.git cd terraform-libvirt-ubuntu-examples/simple
First, create an ssh keypair that can be used when we later want to login to this new VM.
ssh-keygen -t rsa -b 4096 -f id_rsa -C simple -N "" -q
Then use terraform to initialize the configuration defined in “simple.tf”
$ terraform init Initializing the backend... Initializing provider plugins... - Checking for available provider plugins... - Downloading plugin for provider "template" (hashicorp/template) 2.1.2... The following providers do not have any version constraints in configuration, so the latest version was installed. To prevent automatic upgrades to new major versions that may contain breaking changes, it is recommended to add version = "..." constraints to the corresponding provider blocks in configuration, with the constraint strings suggested below. * provider.template: version = "~> 2.1" Terraform has been successfully initialized! You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands should now work. If you ever set or change modules or backend configuration for Terraform, rerun this command to reinitialize your working directory. If you forget, other commands will detect it and remind you to do so if necessary.
Then have terraform show you what it plans on building given the “simple.tf” definition, then have it apply and start constructing.
# show plan terraform plan # apply plan terraform apply # underlying libvirt list of domains virsh list
While terraform is used to setup the virtual hardware resources, cloud-init is used to initialize the OS level configuration including: user accounts, ssh login keys, packages, and networking.
You have to give it a couple of minutes to establish DHCP networking, but after the creation of the VM you can do a refresh, then get a listing of its assigned IP address using:
# have terraform show IP address terraform refresh && terraform output ips # have libvirt show all IP address leases virsh net-dhcp-leases default
If this doesn’t work the first time you try, give the guest OS a couple of minutes to gain network connectivity. Then you can ssh into the VM using the private key:
ssh ubuntu@192.168.122.<lastOctet> -i id_rsa
Other Examples
For further exploration, I have multiple examples in my github repository:
- simple – simple host with DHCP address
- staticip – host with static IP address
- datadisks – host with 2 additional data disks, one xfs and the other ext4
- nestedkvm – passes through cpu features, installed docker and kvm with data drive
- lclone-cinit – linked clone using backing file with cloud-init for customization
- net-install – Ubuntu net installer
- iso-install – Ubuntu fat ISO installer
- linkedclone – linked clone using backing file from ISO installer
REFERENCES
github dmacvicar, Terraform provider libvirt
Terraform, configuration language
ubuntu.com, cloud-init documentation
Ubuntu, UEC/Images/KVMKernelOptions
computingforgeeks.com, How to provision VMs on KVM with terraform
github, terraform-provider-libvirt pull 242 for kernel/initrd/cmdline
emilwypch.com, terraform with additional vsphere disk
github sghuyennet, terraform-vsphere-standalone
blah.cloud, cloud-init and vm templating vmware, cloud-init with kubernetes and docker packages
itzg github, gist on using ubuntu cloud-init image with libvirt
niteshvganesh, full tutorial on terraform on libvirt and azure
cloudinit, debugging by starting with kvm/qemu and cloudinit image
stafwag, centos and cloud-init
kubernetes.io, Installing kubeadm
NOTES
Remove ssh entry, required when host is recreated
ssh-keygen -f ~/.ssh/known_hosts -R <host>
cloudinit logs
/var/log/{cloud-init,cloud-init-output}.log
check DNS settings of systemd-resolv
sudo systemd-resolve --status
View kernel parameters
$ cat /proc/cmdline BOOT_IMAGE=/boot/vmlinuz-4.15.0-74-generic root=UUID=6c7...c7 ro quiet cgroup_enable=memory swapaccount=1
True size of sparse file
qemu-img info /data/kvm/pool/simple1-qcow2
login to VM with private key
ssh ubuntu@192.168.122.30 -i id_rsa
Make sure any old ssh keys are cleared, then re-add, then ssh (helps when rebuilding VM a lot)
ssh-keygen -f ~/.ssh/known_hosts -R "192.168.122.30" && ssh-keyscan -H 192.168.122.30 >> ~/.ssh/known_hosts && ssh ubuntu@192.168.122.30 -i id_rsa
teardown and then rebuild terraform plan
terraform destroy -auto-approve && terraform apply -auto-approve
If network interface is not immediately populated
terraform refresh && terraform output
If issues with permissions on disk creation and permissions on Ubuntu, disable AppArmor in /etc/libvirt/qemu.conf [1,2,3]
security_driver = "none" user = "<yourid>" group = "libvirt" sudo systemctl restart qemu-kvm sudo systemctl restart libvirtd