“pci passthrough” howto

Aims

With this howto you got the chance to set up a Windows virtual machine that is capable of playing video games on a Linux host.
For additional information read “pci passthrough” vs. “vga passthrough” first. Also see the intro post for the hardware I use.

Prepare System

  • activate vt-d in your BIOS
  • set chipset video card to always on
  • install fresh _Ubuntu 12.04 (I chose XUbuntu)
    (For _Ubuntu 12.10 see additional information here.)

Installation

With installing Xen I followed the wonderful Ubuntu Community but I will tell you the important steps right here, too.
(I’m assuming you got root by sudo -i, so “sudo” is left out.)

Install and check the Xen kernel.

apt-get install xen-hypervisor-amd64
sed -i 's/GRUB_DEFAULT=.*\+/GRUB_DEFAULT="Xen 4.1-amd64"/' /etc/default/grub
update-grub
sed -i 's/TOOLSTACK=.*\+/TOOLSTACK="xm"/' /etc/default/xen
xm list

“xm list” should output something like “Domain-0″ representing your Dom0 (host) after a reboot with the Xen kernel.

Set network up. (You could also disable network-manager permanently if you like.)

apt-get install bridge-utils
/etc/init.d/network-manager stop

Change network configuration in /etc/network/interfaces.

auto lo
iface lo inet loopback

auto xenbr0
iface xenbr0 inet dhcp
    bridge_ports eth0

auto eth0
iface eth0 inet dhcp
/etc/init.d/networking restart

Create the volume group “virtual-machines” and add a 100GB volume. (sdb is an empty HDD without any partitions, you may change this as you need.)

vgcreate virtual-machines /dev/sdb
lvcreate -L 100G -n win7x64 virtual-machines

Get the addresses of the pci devices you want to pass through. (Here I’m directly grepping for USB and VGA devices.)
Note: For my HD7950 and other newer video cards you always need to get 2 pci addresses. I found out ’01:00.0′ by the following command and had to add ’01:00.1′ as well to my config file. I assume the second address is responsible for audio stuff.

lspci | grep VGA
lspci | grep USB

Save the following lines as /etc/xen/win7x64-hvm.cfg and change them as you like.

# nut sure if really needed
kernel="/usr/lib/xen-default/boot/hvmloader"

builder = "hvm"
name = "win7x64-hvm"

# RAM size in MB
memory = "4096"

# number CPU cores
vcpus = 2

vif = ['type=ioemu']

# boot from cd and hdd
disk = ['phy:/dev/virtual-machines/win7x64,hda,w','file:/home/user/images/WIN_7_PROFESSIONAL.iso,hdc:cdrom,r']
boot="dc"

# boot from hdd
#disk = ['phy:/dev/virtual-machines/win7x64,hda,w']
#boot="c" 

# enable remote vnc access until video card driver is installed
vnc = 1 

acpi=1
sdl=0
serial='pty'

# list your pci devices (HD7950,HD7950,USB)
pci=['01:00.0','01:00.1','00:1a.0']

Enable pciback module.

modprobe xen-pciback

Create the following bash script to reassign your PCI devices to the virtual machine/pciback and execute it.

remove_device () {
BDF=$1
# Unbind a PCI function from its driver as necessary
[ ! -e /sys/bus/pci/devices/$BDF/driver/unbind ] || \
echo -n $BDF > /sys/bus/pci/devices/$BDF/driver/unbind
# Add a new slot to the PCI Backend's list
echo -n $BDF > /sys/bus/pci/drivers/pciback/new_slot
# Now that the backend is watching for the slot, bind to it
echo -n $BDF > /sys/bus/pci/drivers/pciback/bind
}

#usb controller
remove_device "0000:00:1a.0"
#HD 7950
remove_device "0000:01:00.0"
remove_device "0000:01:00.1"

Create and start your vm.

xm create /etc/xen/win7x64-hvm.cfg

Install vncviewer and connect to the virtual machine.

apt-get install vncviewer

vncviewer localhost:0

Install Windows, video card driver, connect 2nd screen to passed video card and reboot the vm.

I advice you to change the config file and exclude the CD image so the vm is only booting from hdd.

To get the network working you may need to install XEN Windows drivers located here. (additional info)

Results

I don’t like excessive benchmarking but here are some facts about the system.

Team Fortress 2 at 300FPS on High.

Windows7 performance index. (HDD score may be improved by directly passing the SATA controller as well.)
Leistungsindex Windows7 PCI Passthrough XEN

Fixes & Tipps

If you get an error like “device not ready for commands” open the qemu log.
The error “couldn’t find keymaps/en-us” could be fixed by creating the following symlink.

ln -s /usr/share/qemu-linaro /usr/share/qemu

If you are told that a specific pci address has to be assigned to the same guest as another you are probably missing the second pci address of your video card. (see “Get the addresses of the pci devices you want to pass through.” on above)

For the people out there who tried this howto: I’d totally love to hear how it is going and if this howto needs an update.

Links

  • https://help.ubuntu.com/community/XenProposed
  • http://wiki.ubuntuusers.de/Logical_Volume_Manager
  • http://phoronix.com/forums/showthread.php?57068-Xen-VGA-passthrough-is-the-way-to-go!!! (beginning at page 6 or 7)

“pci passthrough” vs. “vga passthrough”

If you have multiple video cards and you want to use one for your Dom0 aka host and the others for your VM, “PCI Passthrough” is the path to take because it is way easier to setup. In conclusion this is what I will try first because I have my HD7950 and the i7 integrated Intel 2000.

On the other hand if you just have one video card (e.g. in a laptop) you have to deal with a little patching and kernel compiling to get your video card to the client. Maybe I will try this later on even though I do not need to.

new project “vga/pci passthrough” – requirements

In my new project “vga/pci passthrough” aka “bye dualboot” I’ll set up a virtual machine using KVM or XEN containing Windows XP or 7 as guest OS under a debian based Linux as host.
Nothing extraordinary new until now, so I add PCI Passthrough for the graphics card, USB controller (keyboard/mouse) and maybe SATA/HDD.

This will hopefully result in a virtual machine on which playing windows games works like a charm and what is easyly managable (backup, copy, move).

There are two ways of realizing this:

  1. IOMMU (AMD)
  2. VT-D (Intel)

I choose Intel’s VT-D technology because they provide awesome VT-D µATX mainboards and I like small desktop setups. In the case space does not matter I would have chosen the AMD FX-8120 with an ATX board because it is awesome for virtualization and way cheaper than the Intel i7 processors in that category.
So this is my setup:

  • CPU: Intel Core i7-2600
  • Mainboard: Intel DQ67SW
  • Graphics: Radeon HD 7950
  • RAM: 16GB Kingston ValueRAM DIMM
    (If you are interested in all of the hardware ask for it and I will tell you.)

Currently I’m sitting here waiting for the hardware to be shipped. So keep up, I’ll post my experiences and how-tos as soon as possible.