PCIe Passthrough and libvirt

A pretty awesome feature of any modern hypervisor is the ability to pass through physical devices like USB and PCIe, without using host drivers. Almost any PCIe device can be passed through, including GPUs. There are many guides online discussing GPU passthrough, including this one by @bryansteiner.

GPUs, however, aren't as easy because they are initialised before the operating system starts. Upon boot, the firmware (sometimes called BIOS) initialises the GPU into VGA mode and starts drawing immediately. Any graphical interface (including the Linux console) will be constantly writing frames to the video card. To free up a GPU for use, you can either go purchase and install an additional GPU or instruct the operating system to stop writing to the current one.

I'm not willing to spend on another GPU on my system, therefore I went the latter route. Most guides online mention using two GPUs, but doing so with one GPU will work too by configuring the system over SSH. I used Fedora Server as the host and libvirtd for virtualization.

I wanted multiple desktop environments, including Fedora Workstation and Windows 10. I started by setting up the VMs using Cockpit, and then added the Nvidia GTX1060 3GB I had to both VM configurations using virsh edit.

With multiple desktop environments, I could shut down one and reboot into the other at any time, like a dual-boot setup, while the host OS is still running essential services. It also allows me to VNC into Fedora while Windows is running by booting Fedora without the GPU.

When I first added the GPU, I noticed that the Linux console was still writing a cursor to the framebuffer while Windows was booting up. It took some digging to figure out that the the framebuffer of the host was still attached. With reference to @joeknock90's scripts, I made an equivalent systemd unit that is run before libvirtd starts. This script ensures that the nouveau drivers are unloaded and the framebuffer for the Linux console is detached.

[Unit]
Description=Disable VTY console for Nvidia GTX 1060 passthrough
Before=libvirtd.service

[Service]
Type=simple
RemainAfterExit=yes
ExecStart=-sh -c "echo 0 > /sys/class/vtconsole/vtcon0/bind && echo 0 > /sys/class/vtconsole/vtcon1/bind && echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind && sleep 1 && virsh nodedev-detach pci_0000_09_00_0 && virsh nodedev-detach pci_0000_09_00_1 && echo 'Successfully detached console and GPU'"
ExecStop=-sh -c "virsh nodedev-reattach pci_0000_09_00_1; virsh nodedev-reattach pci_0000_09_00_0; modprobe nouveau; echo 1 > /sys/class/vtconsole/vtcon0/bind; echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/bind; echo 'Finished attaching GPU and console'"

[Install]
WantedBy=libvirtd.service # Start when libvirtd is started

I also had to apply the usual fix for Error 43 when using Nvidia drivers in Windows.