QEMU PCI Passthrough
WIP This is a work in progress... should still be partially useful.
These instructions are intended for an Alpine Linux installation, the
most recent version at the time of writing was 3.19 (kernel 6.6.5-0-lts).
IMPORTANT Ensure the virtualisation extensions, Intel VT-x or AMD-V are enabled in the BIOS or UEFI setup utility.
IMPORTANT Ensure that Intel VT-d or AMD-Vi is enabled in the BIOS or UEFI setup utility.
IMPORTANT Because of modern UEFI firmware the steps below will probably cause the system to behave somewhat strangely. Configuration of the system should be via serial interface (yeah it will work, Alpine is great) or over ssh provided the network is configured before performing the steps below.
Install Alpine
Install Alpine Linux and configure networking and SSH. Install the following programmes:
apk add qemu-system-x86_64 pciutils
Networking will NOT be covered in detail in this tutorial but iptables or
nftables should probably be installed and configured (sorry, that is
out-of-scope).
apk add iptables
The following packages will make the process easier:
apk add tmux
Configure initramfs and Kernel
Add kernel command line options:
intel_iommu=on iommu=pt kvm.ignore_msrs=1 video=efifb:off
Add modules to initramfs:
vi /etc/mkinitfs/features.d/vfio.modules
kernel/drivers/vfio/*
kernel/drivers/vfio/pci/*
https://wiki.alpinelinux.org/wiki/Initramfs_init:
mkinitfs -c /etc/mkinitfs/mkinitfs.conf -b / <kernelvers>
TODO: Verify that the modules are in the initramfs.
Configure Modules
vfio must be configured to handle the devices and other modules configured
to ignore them... find the modules with lspci:
# lspci -nnk
05:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP107GL [Quadro P400] [10de:1cb3] (rev a1)
Subsystem: NVIDIA Corporation Device [10de:11be]
Kernel driver in use: nouveau
05:00.1 Audio device [0403]: NVIDIA Corporation GP107GL High Definition Audio Controller [10de:0fb9] (rev a1)
Subsystem: NVIDIA Corporation Device [10de:11be]
Kernel driver in use: snd_hda_intel
See the line Kernel driver in use and add these modules to the blacklist
along with the ids of the card that will be used for passthrough:
cat /etc/modprobe.d/local.conf
blacklist nouveau
blacklist snd_hda_intel
options vfio-pci ids=10de:1cb3,10de:0fb9,10de:1cb6,10de:0fb9
options vfio-pci disable_vga=1
After a reboot lspci should show /
Warning!
The kernel option video=efifb:off will probably make the computer difficult
to use locally. I recommend performing all these steps over an ssh connection.
Configure Virtual Machine
Configure QEMU command line to use the devices above.
Stubbing a PCI Device
In cases where the device should not be used by the operating system, for
example an automatically detected on-board graphics card causing issues for
software running on the machine, it may be useful to stub the card using the
pcistub module. See the SO question in references (Stub the device).
pci-stub.ids=<vendorID>:<deviceID>
Since a graphics adapter is a component that is initialised early and serves as one half of the console (one of the most critical portions of a computer system) it may be initialised through other methods such as UEFI. To work around this disable that too:
video=vesafb:off,efifb:off
This will probably cause issues during boot if you have to interact with the computer in early stage boot (such as when asked to provide a password for an encrypted root volume).
Redirecting output to a serial console could also be a possibility, who does not love having to boot a computer using another computer!
TODO: More detail here...