e42.uk Circle Device

 

Quick Reference

Building Alpine Linux VM Images for Terraform

Building Alpine Linux VM Images for Terraform

This will rely on quite a bit of assumed knowledge, please let me know if any of the article can be expanded upon.

Rough Aim

Create a script that will make an Alpine Linux VM image we can build and use in a terraform environment with the terraform libvirt provider.

Prerequisites

Installation should be easy:

apk add terraform go make libvirt-dev libvirt-client libvirt-daemon \
    qemu-img qemu-nbd qemu-system-x86_64
git clone https://github.com/dmacvicar/terraform-provider-libvirt
cd terraform-provider-libvirt
make

For terraform 0.12:

mkdir -p ~/.terraform.d/plugins/linux_amd64/
cp terraform-provider-libvirt \
    ~/.terraform.d/plugins/linux_amd64/terraform-provider-libvirt_v0.0.1

For terraform 0.13 a new mechanism has been introduced explicit provider source locations to allow terraform to find the plugin it must be installed in a namespace:

mkdir -p ~/.terraform.d/plugins/bengreen.eu/corp/libvirt/0.0.1/linux_amd64
cp terraform-provider-libvirt \
    ~/.terraform.d/plugins/bengreen.eu/corp/libvirt/0.0.1/linux_amd64/terraform-provider-libvirt_v0.0.1

The next step is to check that terraform can find the plugin...

Check Terraform can find terraform-provider-libvirt

Fairly important, create a directory tftest then inside create a testlv.tf file.

provider "libvirt" {
    uri = "qemu:///system"
}
resource "libvirt_domain" "terraform_test" {
    name = "terraform_test"
}

For terraform 0.13 another directive is required to tell terraform where the plugin is located, add this to the testlv.tf file, it can be at the top or bottom (there is nothing special about bengreen.eu/corp/libvirt it is just to match the directory where the plugin can be found):

terraform {
    required_providers {
        libvirt = {
            versions = ["0.0.1"]
            source = "bengreen.eu/corp/libvirt"
        }
    }
}

Then run terraform init, the output should have a line in green like this:

Terraform has been successfully initialized!

If that does not work check that the file in ~/.terraform.d/plugins/linux-amd64/ is named correctly. This command should work even though libvirtd is not running.

QEMU (and QEMU/KVM)

libvirt and consequently the libvirt terraform provider use QEMU as the underlying virtualisation technology and so it helps to understand the usage of QEMU for testing and when things go wrong (things always go wrong).

To build the virtual machine image a tool that has been written by the Alpine Linux project will be used called alpine-make-vm-image. It makes the process much easier than by hand.

The script uses ext-linux from a chroot and will fail with a line in dmesg about CAP_SYS_RAWIO unless the kernel is told to allow it:

sysctl -w kernel.grsecurity.chroot_caps=0

First, decide which packages to install, for this example openssh and qemu-guest-agent are required. A script can be written ./alpinevmbase.sh that will be executed in the chroot of the new VM image (--script-chroot is important!) this script will be used to perform one-time configuration (it is not executed each time the VM is started):

./alpine-make-vm-image --image-format qcow2 --image-size 16G \
    --packages "openssh qemu-guest-agent" \
    --serial-console \
    --script-chroot alpinevmbase.qcow2 ./alpinevmbase.sh

The image is now built and available as alpinevmbase.qcow2 to keep the image as built use this file as the base for the testing image:

qemu-img create -b alpinevmbase.qcow2 -f qcow2 os-base.qcow2

Be sure the tun module is installed to connect to the network interface:

modprobe tun

And finally, start the VM:

qemu-system-x86_64 \
    -machine pc-q35-2.10 \
    -net nic,model=virtio,macaddr=00:00:CA:FE:BA:BE \
    -net tap,ifname=tap11,script=no,downscript=no \
    -vnc 127.0.0.1:11 \
    -serial tcp:localhost:6011,server,nowait \
    -drive if=virtio,file=os-image.qcow2,format=qcow2,discard=unmap \
    -chardev socket,path=/tmp/qga.sock,server,nowait,id=qga0 \
    -device virtio-serial \
    -device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0

Connecting to the VNC port or the console serial interface with netcat should present a login prompt (press enter if nothing appears):

nc 127.0.0.1 6011

The password for root is not set.

Connect with pty

Use the following arguments:

-chardev pty,id=charserial0 \ -device isa-serial,chardev=charserial0,id=serial0

Then connect with socat:

socat open:/dev/pts/8 readline

See socat manual page for details of raw,crnl options, they may be required.

Test the QEMU Guest Agent

To test the Guest Agent socat will connect to the socket created by qemu at /tmp/qga.sock and a JSON encoded command may be sent and terminated by a new line.

sudo socat unix-connect:/tmp/qga.sock readline
{"execute":"guest-sync", "arguments":{"id":1234}}
{"return": 1234}

libvirt will connect to the guest this way to discover the IP address and many other things (files etc can be created and edited this way, see Communicate with QEMU Guest Agent in references).

Show the network interfaces on the guest, including MAC and IP addresses:

sudo socat unix-connect:/tmp/qga.sock readline
{"execute":"guest-network-get-interfaces"}
{... lots of stuff ...}

Terminate the VM, poweroff or Ctrl-C. os-image.qcow2 is not required anymore so it may be deleted.

There are may references to the qemu-guest-agent using /dev/virtio-ports/org.qemu.guest_agent.0 I think this is probably made possible with udev (or eudev) which renames system devices like eth0 etc. Alpine Linux does not perform this rename operation by default so to use the QEMU agent the path is /dev/vport1p1 or /dev/vport2p1. This change can be seen in the build script alpinevmbase.sh.

libvirt

The problem with the above configuration is that there is no DHCP server on tap11 so querying the IP addresses is fairly pointless. The IP address may be set manually but surely the whole point of this is automation. libvirt is a wrapper around qemu (and other virtualisation-like technologies) which, when correctly configured, will run a DHCP server for the virtual machine to acquire an IP address from.

In the same way as QEMU was demonstrated without any kind of libvirt interference this section will demonstrate the use of libvirt without any terraform interference!

Start the libvirt-daemon:

/etc/init.d/libvirtd start

libvirt pools

Storage in libvirt is in a pool, the simplest pool type is a directory and can be defined with the virsh tool. The default location seems to be /var/lib/libvirt/images but this is not defined in Alpine Linux so must be defined manually and activated:

virsh pool-define-as default dir --target /var/lib/libvirt/images
Pool default defined
virsh pool-start default

To use the virtual machine image it must be imported into the pool, first a volume must be created in the pool:

virsh vol-create-as default alpinebase.qcow2 16G --format qcow2

Once the volume is created it should be visible as a file in /var/lib/libvirt/images the image created earlier should now be imported:

virsh vol-upload alpinebase.qcow2 alpinevmbase.qcow2 --pool=default

The alpinebase.qcow2 image could be used directly by the virtual machine but as in the previous example the base image should be reusable and all changes should be stored separately:

virsh vol-create-as default os-image.qcow2 16G \
    --format qcow2 \
    --backing-vol /var/lib/libvirt/images/alpinebase.qcow2 \
    --backing-vol-format qcow2

This makes a bit of a mockery of the whole pool idea, this is because the qemu way of doing things is leaking through (remember libvirt in this instance is a wrapper around the underlying qemu-img command used earlier). There may be some way of defining a backing volume by referring to a pool... I don't know. The default argument referrs to the destination pool.

libvirt networking

So far this tutorial has avoided XML file editing but no longer, libvirt configuration is all XML based and so far it has been hidden by the commands used.

To allow the machines to do anything useful they will need to communicate with the outside world. The XML below will create a simple network, alter the addresses to taste:

<network>
  <name>test-network</name>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr-test' stp='on' delay='0'/>
  <ip address='192.168.200.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.200.2' end='192.168.200.254'/>
    </dhcp>
  </ip>
</network>

Example taken from Managing KVM with libvirt, see references.

Save this file as test-network.xml and use virsh to define and start it:

virsh net-define test-network.xml
virsh net-start test-network

In the alpinevmbase.sh file the network was configured to use DHCP now the environment is appropriately configured to provide the DHCP capability to the guest.

IPv6 Connectivity with HE.net

The additional routing specification required for IPv6 routing the network provided by HE.net is:

<ip family="ipv6" address="2001:db8:ca2:2::1" prefix="48"/>

Routing must be correctly configured on the host and the host must be connected via an IPv6 tunnel to HE.net. This is a little out of scope for this tutorial.

libvirt domains

To bring the volume and the network interface together a domain must be created. As with the network definition the domain is defined with XML this XML file is quite large and so I have not included it here, see the github repository for details (test-instance.xml).

virsh define test-instance.xml
virsh start test-instance

Virsh can find the IP address of the started instance via the lease provided from the dnsmasq DHCP server, to discover the IP:

virsh domifaddr test-instance

To discover the IP address via the qemu-guest-agent:

virsh qemu-agent-command test-instance '{"execute":"guest-network-get-interfaces"}'

Depending on specific setup of the test-instance.xml document the qemu-guest-agent may not be operating correctly. In this case the previous command will return an error:

 error: Guest agent is not responding: QEMU guest agent is not connected

To correct this problem the correct entry in /dev must be chosen for the qemu-guest-agent configuration file. To discover the correct device (it is a virtio-serial character device) enter the VM console:

virsh console test-instance

Press enter if nothing happens and login as root (there is no password). The vm image was configured earlier to use /dev/vport1p1 in /etc/conf.d/qemu-guest-agent:

cat /etc/conf.d/qemu-guest-agent
GA_METHOD="virtio-serial"
GA_PATH="/dev/vport1p1"

Change this to the device found in /dev (probably /dev/vport2p1) then re-start the guest agent:

echo -e 'GA_METHOD="virtio-serial"\nGA_PATH="/dev/vport2p1"' > /etc/conf.d/qemu-guest-agent
/etc/init.d/qemu-guest-agent restart

To quit from this console type Ctrl+] and try the qemu-agent-command again this time it should succeed.

Using qemu-guest-agent to shutdown

Other methods are available and in a modern cloud it is preferable to create services that do not need to be shutdown, never the less the agent can be used to shut down the system if desred:

virsh shutdown --mode agent test-instance
Last updated 26 September 2020

Cleaning up ready for Terraform

The last sections serve as a behind the scenes look at what is going on when terraform creates and updates libvirt VMs, now the workspace will be cleaned up to allow terraform to perform a lot of the gruntwork.

...

References

Last updated 2024-04-15

Quick Links: Techie Stuff | General | Personal | Quick Reference