linux

PC Build "Sinon" : The best work/gaming rig

PCI Passthrough -> The real gaming on linux

Maxence Sartiaux

• 8 min read
PC Build "Sinon" : The best work/gaming rig

Disclaimer

This is an evolutive post, it might change from time to time, i'm using this setup everyday, some update may break some configuration you'll find in this post.

If you find an error while you read it, don't hesitate to send me an e-mail to contact+blog at makz.me

Introduction

For less than one year now, i use my "D.Va" build, my main OS is Archlinux and i have a dualboot for playing games on Windows. Unfortunately this is not convenient at all for me, everytimes when i'm on windows playing a game, i have to connect to my server or fix a bug in a piece of code and i have to reboot, if i listen to myself i could reboot 5 times per days, and i'm used to never turn off my computer so sometime i'm on windows for two weeks and it's two weeks i don't work at home, the other time it's 2 weeks on Arch and i can't play a single game ...

Here, we have a dramatic case of first world problem.

So i decided to exeperiment the gaming in a virtual machine !

My idea was to pass my GTX 1080 to a windows virtual machine and play on this one, after some tests with spare graphic card i have and it's successful,i have almost the same perfomances as a native windows !

So i decided to mod my "D.Va" build and definitively implement this setup on my main rig.

I ran into some problems and if you want to reproduce the same setup, here's the first needs for this setup :

- The GPu you want to passthrough don't have to be your "Primary GPU", in some bios you can select which GPU as "Primary", otherwise you have to change the position of you GPU (my case)
- For better performances, you need a "UEFI compatible" GPU, Check here if your GPU have the "UEFI support", this could work without the UEFI but you have to use the default qemu BIOS.
- Your CPU / Motherboard need to be VT-d (and VT-x indeed) compatible.- Having a pretty good amount of RAM and a good CPU (indeed)

My principal problem was the notion of "Primary GPU", my motherboard don't let me choose which GPU will be selected as "Primary" so i had to move my graphic card to another PCI-e connector, Unfortunately i have a custom watercooling and it's not this easy ^^, i bought a second hand GTX980 to use it as my Primary GPU on Archlinux, moved my 1080 to the second 16x connector and mod my watercooling to have something fully integrated.

Okay once you have a correct hardware we can start !

Hardware

- ASUS X99 Strix
- Intel i7-6900K @ 4.3Ghz
- 8x8Go DDR4 2133Mhz
- GTX980 "Primary GPU"
- GTX1080 Strix "Passthrough GPU"
- HyperX 3K (Archlinux System)
- Samsung 840 Pro 256Go (Windows System)
- Seagate Firecuda 2To (Archlinux Data)
- Seagate Firecuda 2To (Windows Data/Games)

BIOS Settings

- Enable VT-x
- Enable VT-d
- If possible set your GPU priority (Primary = Other than the pass-through GPU, dedicated to linux)

Screen connections

Because you use another GPU, you need to connect your screen to the second GPU.

the idea in the end, is when you start your VM, it disable your main screen in linux, the VM start, your screen detect the input from the second GPU and autoswitch to this input, when the VM is down, you reactive your screen in linux and the screen switch back.

For my setup i've complexified the process, it'll be explained later in this article.

So it'll look like that

For the first setup i recommend using a second screen (one per GPU) it'll be easier in case of problems.

Installing requirements

Install virt-manager & qemu

yaourt -Sy virt-manager qemu

If your Passthrough GPU have the UEFI support install the OVMF bios from AUR

yaourt -S ovmf-git

Isolating the GPU


First of all you need to enable IOMMU


Add intel_iommu=on (or amd_iommu=on if you have a AMD CPU) in the grub cmd line (/etc/default/grub)

...
GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on quiet"
...

Update your grub.cfg

grub-mkconfig -o /boot/grub/grub.cfg

Reboot and ensure if IOMMU is correctly enabled

dmesg|grep -e DMAR -e IOMMU


[    0.000000] ACPI: DMAR 0x00000000BDCB1CB0 0000B8 (v01 INTEL  BDW      00000001 INTL 00000001)
[    0.000000] Intel-IOMMU: enabled
[    0.028879] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a
...

We are good, now enable vfio

Find your the ID of your GPU

lspci -nnk | grep -i nvidia

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF119M [NVS 4200M] [10de:1056] (rev a1)
	Kernel driver in use: nvidia
	Kernel modules: nouveau, nvidia_drm, nvidia
01:00.1 Audio device [0403]: NVIDIA Corporation GF119 HDMI Audio Controller [10de:0e08] (rev a1)

Here we can see my ID is 10de:1056 & 10de:0e08

vim /etc/modprobe.d/vfio.conf

options vfio-pci ids=10de:13c2,10de:0fbb

And we load the kernel module at boot

vim /etc/mkinitcpio.conf

MODULES="... vfio vfio_iommu_type1 vfio_pci vfio_virqfd ..."

Rebuild the initrd

mkinitcpio -p linux

And reboot.Check if the module is correctly loaded

dmesg | grep -i vfio

[    0.329224] VFIO - User Level meta-driver version: 0.3
[    0.341372] vfio_pci: add [10de:13c2[ffff:ffff]] class 0x000000/00000000
[    0.354704] vfio_pci: add [10de:0fbb[ffff:ffff]] class 0x000000/00000000
[    2.061326] vfio-pci 0000:06:00.0: enabling device (0100 -> 0103)

And if the GPU is correctly isolated, with this command you should see if the driver in use is "vfio-pci"

lspci -nnk -d 10de:13c2

06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)
	Kernel driver in use: vfio-pci
	Kernel modules: nouveau nvidia

If this method is not working you could use the old method "pci-stub", but i didn't use it, refer to the archlinux wiki

Okay, now our GPU is ready, time to create the VM !

The virtual machine

At first we need to configure libvirtd to tell him to use the OVMF bios

/etc/libvirt/qemu.conf

nvram = [
	"/usr/share/ovmf/x64/OVMF_CODE.fd:/usr/share/ovmf/x64/OVMF_VARS.fd"
]

Then simply create with virt-manager a vm, don't forget to check "Customize before install" select the UEFI firmware.  
CPU Section, set the model "host-passthrough"

Thoses are basic settings, you may want to tune your VM for your own setup.

Once the VM is create, add PCI devices (basicaly your GPU & your USB controller (I recommend to not add the USB controller at first or the VM will grab your keyboard/mouse and if you get in trouble you'll have to find a way to shutdown the VM without keyboard (SSH))).

For your first start, I recommend to use a dedicated screen on the second GPU and other keyboard/mouse only for the VM.

Pass your main screen to Windows

Your Windows is ready to rock ? Nice.

The idea is to remove your spare display used for the installation and connect the second input of your main display to the second GPU.  Then we create a qemu hook in libvirt which disable the linux primary display, this will force your display to go into "Auto-Input mode" and it'll switch to the second GPU input once the VM GPU is initialized.

Here's my hook script, as you can see at first i used xrandr to turn off/on the primary screen and adapt my resolution but since few updates, using xrandr generate a segfault into gnome-shell and it kill my graphical interface "No good"  So i end up using nvidia-settings.

vim /etc/libvirt/hooks/qemu

#!/bin/bash

export DISPLAY=:1
export XAUTHORITY=/run/user/1000/gdm/Xauthority

if [ "$1" == "win10" ]; then
	if [ "$2" == "start" ]; then
		#xrandr --output DVI-I-1 --off --output DP-2 --mode 1920x1080 --pos 0x0 --rotate normal --output HDMI-0 --mode 1920x1080 --pos 1920x0 -r 60 --rotate normal
		nvidia-settings --assign CurrentMetaMode="DPY-5: 1920x1080 +0+0 {ViewPortIn=1920x1080, ViewPortOut=1920x1080+0+0}, DPY-2: 1920x1080 +1920+0 {ViewPortIn=1920x1080, ViewPortOut=1920x1080+0+0}"
	fi
	if [ "$2" == "stopped" ]; then
		#xrandr --output DP-2 --mode 1920x1080 --pos 0x0 --rotate normal --output DVI-I-1 --mode 1920x1080 -r 60 --pos 1920x0 --rotate normal --output HDMI-0 --mode 1920x1080 --pos 3840x0 -r 60 --rotate normal
		nvidia-settings --assign CurrentMetaMode="DPY-1: 1920x1080 +1920+0 {ViewPortIn=1920x1080, ViewPortOut=1920x1080+0+0}, DPY-5: 1920x1080 +0+0 {ViewPortIn=1920x1080, ViewPortOut=1920x1080+0+0}, DPY-2: 1920x1080 +3840+0 {ViewPortIn=1920x1080, ViewPortOut=1920x1080+0+0}"
	fi
fi

Install the AC97 driver on Windows 10

You'll only have a good sound with the hardware virtualisation but Windows 10 removed the AC97 support so we have to trick him, you need to boot with disabled driver signature and then Windows will let you install AC97 driver.

Check this post for more informations.

Pass the sound to the host

To connect the qemu sound to pulseaudio, you'll have to edit the libvirtd/qemu configuration

/etc/libvirt/qemu.conf

# Replace with your username
user = "makz"

Then edit the domain key in the libvirt VM XML file

virsh edit win10

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>

And finally add the QEMU PulseAudio environment variables at the end (edit the user id if needed)

</devices>
	<qemu:commandline>
		<qemu:env name='QEMU_AUDIO_DRV' value='pa'/>
		<qemu:env name='QEMU_PA_SERVER' value='/run/user/1000/pulse/native'/>
	</qemu:commandline>
</domain>

Share your keyboard & mouse

To avoid input lag we passthrough the Keyboard/Mouse or directly the USB controller but that mean, your linux is unusable while Windows is running.  
The idea here is to use a mouse/keyboard sharing software like Synergy and set the Windows as server and linux as client.

Exchange a linux screen to a second Windows screen

Everyone who already worked with more than one screen know one is not enough !

So the final step is to add the possibility to switch from "2 linux screens - 1 windows" to "1 linux screen - 2 windows".

I did that with a simple KVM switch and a simple script in bash that check every 5 seconds if my "switched" screen is present on linux, if it's not, it change the geometry to enable just one screen, and vice-versa.

This is the physical setup

cat /usr/local/sbin/3to2

#!/bin/bash

export DISPLAY=:1
export XAUTHORITY=/run/user/1000/gdm/Xauthority

echo "The magic is running"

function loop {
	virsh -c qemu:///system 'dominfo win10' | grep running > /dev/null 2> /dev/null
	if [ $? -eq 0 ]; then
		ARRAY=$(nvidia-settings --query all | grep 1920 | grep -o '[A-Z]..-[0-9]:')
		echo $ARRAY | grep DPY-5 > /dev/null 2> /dev/null
		if [ $? -ne 0 ]; then
			echo $ARRAY | grep DPY-1 > /dev/null 2> /dev/null
			if [ $? -eq 0 ]; then
				nvidia-settings --assign CurrentMetaMode="DPY-2: 1920x1080 +0+0 {ViewPortIn=1920x1080, ViewPortOut=1920x1080+0+0}" > /dev/null 2> /dev/null
				echo "Changing to one screen X X O"
			fi
		else
			echo $ARRAY | grep DPY-1 > /dev/null 2> /dev/null
			if [ $? -eq 0 ]; then
				nvidia-settings --assign CurrentMetaMode="DPY-5: 1920x1080 +0+0 {ViewPortIn=1920x1080, ViewPortOut=1920x1080+0+0}, DPY-2: 1920x1080 +1920+0 {ViewPortIn=1920x1080, ViewPortOut=1920x1080+0+0}" > /dev/null 2> /dev/null
				echo "Changing to two screen O X O"
			fi
		fi
	fi

}

while true; do loop; sleep 5; done

This script is running as a service with systemd.

Optimized kernel for performances

The more ram your VM has, the longer it take to boot the VM, you'll see once you boot it, all CPU thread goes to 100% for few seconds/minutes then the VM boot.

This is because the arch kernel is Pre-emptive by default.

To fix that, you can rebuild the archlinux kernel to disable PREEMPT, enable PREEMPT_VOLUNTARY and set it to 1000Hz

I used to maintain a kernel repository but as of 2022, i no longer use this setup so my repository is dead. If anyone want more information on the rebuild, i can help you, just contact me :)

Sources

https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF