“pci passthrough” howto

Aims

With this howto you got the chance to set up a Windows virtual machine that is capable of playing video games on a Linux host.
For additional information read “pci passthrough” vs. “vga passthrough” first. Also see the intro post for the hardware I use.

Prepare System

  • activate vt-d in your BIOS
  • set chipset video card to always on
  • install fresh _Ubuntu 12.04 (I chose XUbuntu)
    (For _Ubuntu 12.10 see additional information here.)

Installation

With installing Xen I followed the wonderful Ubuntu Community but I will tell you the important steps right here, too.
(I’m assuming you got root by sudo -i, so “sudo” is left out.)

Install and check the Xen kernel.

apt-get install xen-hypervisor-amd64
sed -i 's/GRUB_DEFAULT=.*\+/GRUB_DEFAULT="Xen 4.1-amd64"/' /etc/default/grub
update-grub
sed -i 's/TOOLSTACK=.*\+/TOOLSTACK="xm"/' /etc/default/xen
xm list

“xm list” should output something like “Domain-0″ representing your Dom0 (host) after a reboot with the Xen kernel.

Set network up. (You could also disable network-manager permanently if you like.)

apt-get install bridge-utils
/etc/init.d/network-manager stop

Change network configuration in /etc/network/interfaces.

auto lo
iface lo inet loopback

auto xenbr0
iface xenbr0 inet dhcp
    bridge_ports eth0

auto eth0
iface eth0 inet dhcp
/etc/init.d/networking restart

Create the volume group “virtual-machines” and add a 100GB volume. (sdb is an empty HDD without any partitions, you may change this as you need.)

vgcreate virtual-machines /dev/sdb
lvcreate -L 100G -n win7x64 virtual-machines

Get the addresses of the pci devices you want to pass through. (Here I’m directly grepping for USB and VGA devices.)
Note: For my HD7950 and other newer video cards you always need to get 2 pci addresses. I found out ’01:00.0′ by the following command and had to add ’01:00.1′ as well to my config file. I assume the second address is responsible for audio stuff.

lspci | grep VGA
lspci | grep USB

Save the following lines as /etc/xen/win7x64-hvm.cfg and change them as you like.

# nut sure if really needed
kernel="/usr/lib/xen-default/boot/hvmloader"

builder = "hvm"
name = "win7x64-hvm"

# RAM size in MB
memory = "4096"

# number CPU cores
vcpus = 2

vif = ['type=ioemu']

# boot from cd and hdd
disk = ['phy:/dev/virtual-machines/win7x64,hda,w','file:/home/user/images/WIN_7_PROFESSIONAL.iso,hdc:cdrom,r']
boot="dc"

# boot from hdd
#disk = ['phy:/dev/virtual-machines/win7x64,hda,w']
#boot="c" 

# enable remote vnc access until video card driver is installed
vnc = 1 

acpi=1
sdl=0
serial='pty'

# list your pci devices (HD7950,HD7950,USB)
pci=['01:00.0','01:00.1','00:1a.0']

Enable pciback module.

modprobe xen-pciback

Create the following bash script to reassign your PCI devices to the virtual machine/pciback and execute it.

remove_device () {
BDF=$1
# Unbind a PCI function from its driver as necessary
[ ! -e /sys/bus/pci/devices/$BDF/driver/unbind ] || \
echo -n $BDF > /sys/bus/pci/devices/$BDF/driver/unbind
# Add a new slot to the PCI Backend's list
echo -n $BDF > /sys/bus/pci/drivers/pciback/new_slot
# Now that the backend is watching for the slot, bind to it
echo -n $BDF > /sys/bus/pci/drivers/pciback/bind
}

#usb controller
remove_device "0000:00:1a.0"
#HD 7950
remove_device "0000:01:00.0"
remove_device "0000:01:00.1"

Create and start your vm.

xm create /etc/xen/win7x64-hvm.cfg

Install vncviewer and connect to the virtual machine.

apt-get install vncviewer

vncviewer localhost:0

Install Windows, video card driver, connect 2nd screen to passed video card and reboot the vm.

I advice you to change the config file and exclude the CD image so the vm is only booting from hdd.

To get the network working you may need to install XEN Windows drivers located here. (additional info)

Results

I don’t like excessive benchmarking but here are some facts about the system.

Team Fortress 2 at 300FPS on High.

Windows7 performance index. (HDD score may be improved by directly passing the SATA controller as well.)
Leistungsindex Windows7 PCI Passthrough XEN

Fixes & Tipps

If you get an error like “device not ready for commands” open the qemu log.
The error “couldn’t find keymaps/en-us” could be fixed by creating the following symlink.

ln -s /usr/share/qemu-linaro /usr/share/qemu

If you are told that a specific pci address has to be assigned to the same guest as another you are probably missing the second pci address of your video card. (see “Get the addresses of the pci devices you want to pass through.” on above)

For the people out there who tried this howto: I’d totally love to hear how it is going and if this howto needs an update.

Links

  • https://help.ubuntu.com/community/XenProposed
  • http://wiki.ubuntuusers.de/Logical_Volume_Manager
  • http://phoronix.com/forums/showthread.php?57068-Xen-VGA-passthrough-is-the-way-to-go!!! (beginning at page 6 or 7)

33 thoughts on ““pci passthrough” howto

  1. Thanks for the tutorial! It’s great to see that Xen PCI passthrough works with Ubuntu (and therefore most likely with other Ubuntu-based distros).

    I am currently trying to get VGA passthrough working on Linux Mint 13 Mate, having a Nvidia Quadro 600 GPU, but I can’t get xen-pciback load as module BEFORE the video card driver (nouveaux). I’m not even close to get the Nvidia card passed through, which may require reading the video card BIOS and compile patches into the kernel or xen.

    So, if I understand you correctly, with two video cards I could use PCI passthrough with much less trouble than VGA passthrough? I.e. I would not need to compile xen-pciback into the kernel (or try figure out a way to make it load and grab the video card before it’s taken by the video driver), nor would I have to worry about Nvidia cards being capable/incapable of VGA passthrough?

    Is the investment into a second video card worth it? My screen has 2 DVI inputs.

    P.S.: Read your posts on the other forum.

    • Hi powerhouse,

      I’m glad you like my tutorial and yes, a second graphics card would help you around VGA passthrough.
      Normally you don’t need that much GPU power under Linux operating systems if you just do some office, coding or web browsing stuff while having the windows VM running. In conclusion I advice you to get a second graphics card that fits this needs and pass the powerful Nvidia graphics card to your Windows VM.
      If you don’t have your Windows VM running you can still use the powerful Nvidia Quadro on your Linux system.

      I share your experience that getting VGA passthrough to work is much more painful than passing just PCI devices. No kernel compiling there.

      Hopefully I could help and feel free to ask further questions.

  2. Thanks for your reply and suggestions!

    I’m contemplating in getting another VGA card. I have a Nvidia 8600GT passive cooled card from my old PC which I could use. But, so far I haven’t been able to run the Quadro 600 card using the proprietary driver under Xen. It works fine on a regular (non-Xen) boot. Nvidia doesn’t support Xen, so I was thinking of getting a low-end AMD 6450 card for dom0 use, in the hope it would work good with the open source drivers and/or with the Catalyst driver. Unfortunately the nouveaux driver for Nvidia lacks many features of the proprietary driver and doesn’t support 3D.

    Any thoughts about mixing Nvidia and AMD on one system? What do you use for your dom0, isn’t it a 6450? Did you get the Catalyst driver work under dom0? Or is the open source driver good enough?

    Many questions, sorry for that.

    One suggestion for your how-to: List your hardware as this is very relevant.

    • I’m sorry for the missing hardware list. You can find it in my previous post, it is now linked in this post above, too.

      I personally don’t use the AMD graphics card under dom0 because the integrated i7 Intel GPU is enough for me, therefore I didn’t install any Catalyst drivers.

      I do not have any experiences in using Nvidia and AMD cards parallely in this setup but as I said, I use the Intel GPU under dom0 and the AMD GPU under domU, so in conclusion I assume that mixing Nvidia and AMD cards should not be a problem.

      • I bought a cheap Sapphire AMD 6450 video card today, as well as a KVM switch to connect the USB keyboard and mouse to two different USB ports residing on different host controllers.

        My motherboard is an Asus Sabertooth X79 so I had to install a USB panel on the back and connect it to the USB1314 header on the board (all Intel X79 chip USB ports on the back panel are under the same host controller, and the other chip is a USB3.0 chip that AFAIK isn’t supported out of the box, neither Windows nor Linux).

        Next I removed my Nvidia Quadro 600 card from the PCI-e 16 slot 1 and placed it in the second slot. I installed the new AMD graphics adapter in the first slot.

        Reboot into Xen and voila – it even recognized my new AMD graphics card and booted right into the desktop. Man, I’ve never seen such a thing in my life! No command line driver removal or install! It just works.

        More than that, I was able to install the proprietary AMD (Catalyst) driver right into the Xen system, reboot and it worked. Just select the default driver (the post-release didn’t work, I tried it).

        I had a few giggles when trying to run the VM, as I used sda instead of hda for the Windows hard drive designation. Seems like Windows doesn’t support SATA without extra driver (which I hadn’t found, perhaps I should have made the Windows installer search the DVD ISO image). Anyway, I changed it to hda and it worked.

        The only thing I forgot to buy is a second DVI cable, so for now I’m just accessing the Windows VM via virt-manager vncviewer.

        Next came the graphics driver for Windows. Went to the Nvidia website, went into their “check for graphics card driver” java utility and it found my Quadro card. Downloaded the newest driver and installed. Windows complained about problems (since there is no screen connected – missing cable). But I went on and am now installing a zillion Windows updates.

        So, I’m really close to report total success. Have to wait until after the weekend to get the DVI cable.

        One note about your how-to:

        ln -s /usr/share/qemu-linaro/keymaps/en-us /usr/share/qemu/keymaps/en-us

        didn’t work and complained about missing /usr/share/qemu/keymaps/en-us.

        I had to change it to:

        ln -s /usr/share/qemu-linaro /usr/share/qemu

        and it worked. Thanks for this advice!

        You really helped me better understand PCI passthrough versus VGA passthrough. I always thought that video cards would only run using VGA passthrough. It’s absolutely not clear on the Xen web sites.

        Thanks for this tutorial – it finally got my system running.

        P.S.: I will post a final note when everything is in place.

        I got Win 7 installed now as a domU.

        • I’m so glad your setup is working now.
          If I helped you to get along, my tutorial is already worth it. Hopefully more follow :)

          Looking foreward to your final note.

  3. I was celebrating too soon :-( .

    While Windows recognizes the graphics card (Nvidia Quadro 600), I get a yellow triangle and error 43 and it won’t use the card.

    I first tried with the latest driver, now I rolled back to the latest 275.xx series driver as suggested in some places.

    When running the domU, the graphics card is assigned to pciback. When closing the guest the PCI devices become assignable.

    I use vpci instead of passthrough, as passthrough=1 in the module options gets me stuck at the boot, after graphical login (black screen with cursor, nothing works not even remote ssh user@computer). Anyway, I was hoping that vpci would work. Any thoughts on that? Or on the Windows driver issues?

    • Is passthrough enabled automatically or did you set it manually?
      Sadly I don’t know anything about vpci but I read about pcistub as an alternative to pciback, maybe that’s useful to you.

      Thanks for the advice concerning the keymaps-error, I updated the howto and replaced the not working command with yours.

      • Passthrough is enabled via script (I’ve adopted another similar script as yours to load as an init script and it works nicely, running at run-level 2-5). Once I’m in the system I manually xm create /dev/xen/win7.cfg and run virt-manager to load Windows, currently using the vncviewer to access Windows as I got now display for it.

        As I said, Windows boots fine and everything seems to be OK (CPUs, memory, etc.), except the Nvidia Quadro card which it recognizes but is unable to initialize, neither with a 305.x driver nor with an older 275.x driver. The Nvidia card isn’t even initialized, as my monitor complains that it doesn’t get an input signal from the card.

        The way it looks like I won’t get around compiling some Xen patches for Nvidia cards. The other option is to switch cards and use my AMD 6450 card for passthrough. I was hoping to avoid this, since the last time I had to compile anything was ages ago.

        Any idea what the status of Xen 4.2 is? It’s supposed to provide lots of improvements but I haven’t found yet any ready-to-go source tree (i.e. without applying additional patches manually). Do you know where to get the latest Xen 4.2 with all patches applied?

  4. Greetings,
    nice tutorial! I will try this this week with an ASUS N56VZ Notebook, which has 2 graphic cards (NVIDIA and Intel HD).

    I want to passtrugh the NVIDIA card to get some good game experience with a Windows VM. Do you have made some experience while doing this with a notebook or at least a notebook hardware?
    The latest things about that, what I found on the internet was, that it is actually possible to do that … but anyway, I just have to try this on myself ;)

    • Welcome ReCon and thank you a lot.

      I love the idea you have and do not have any experience with mobile hardware and PCI passthrough. Although I’m not quite sure if it’s going to work because don’t you need seperate video outputs from the different GPUs?
      Are the graphic cards switched to the outputs or is there one output for onboard and one for the dedicated graphics card?

      Please inform us about your progress here, I would be really excited to see this work on a notebook. Sadly I don’t got the hardware to try it myself…

      • Well, my first try with the ASUS N56VZ was just a pain (useing Ubuntu 12.04). No LED keyboard backlight, Bumblebee doesn’t work and so on.

        Anyway, after this long nights of testing I found out, that the HDMI connector does not work on anytime under Linux. So I was going to research the web for this problem and found out, that the HDMI connector is restrictly bound to the NVIDIA card, while the VGA output is restrictly bound to the Intel HD card.
        So I think, that it should be possible for me to get a video output over HDMI output, while I’ll starting the VM up (with PCI Passtrugh enabled).

        Would you “technically” agree to my thinking, too?

          • Ok, so I will try this later.

            Some people had written, that a consumer NVIDIA card (this means every series expect the quadro once) is needing a custom XEN kernel with a lot of hacks to get Passtrughed. Does this only fit on the VGA Passtrugh or on PCI Passtrugh scenario as well? I didn’t found anything about that, because the most questions to Passtrugh NVIDIA cards at the XEN mailing lists are specific to VGA Passtrugh.

  5. I’m one of those trying to get a Nvidia secondary display adapter passed through (see my previous posts here). Right now I’m compiling Xen 4.2 in the hope it will improve things. If that doesn’t do, I’ll try to extract the VGA BIOS and follow the instructions at http://www.davidgis.fr/blog/index.php?2011/12/07/860-xen-42unstable-patches-for-vga-pass-through in the hope that this will finally work (see also http://www.davidgis.fr/blog/index.php?2012/07/10/917-xen-vga-passthrough-new-patches-for-nvidia for a more updated patch).

    Good luck!

  6. Ok, so now I was able to try this. It worked really well until to the point where I want to fire the VM up.
    Again, my Notebook is a ASUS N56VZ. I’ve readed a lot, that ASUS just only support at nearly 50% of their Desktop Mainboards the VT-d technology, but well, it looks like that notebooks are not able to handle it, too.
    The error message when I’m trying to start the VM:
    root@Rausgelost:~# xm create /etc/xen/win7x64-hvm.cfg
    Using config file “/etc/xen/win7x64-hvm.cfg”.
    Error: failed to assign device: maybe the platform doesn’t support VT-d, or VT-d isn’t enabled properly?

    I know that the “normal” VT technology is available and active, because I have hardware accleration support in VirtualBox, but it seems, that there is no chance for PCI Passthrough, due to the lack of VT-d (there is no option in my BIOS to activate it).
    Sadly, but it looks like that I will stuck into dual boot and WINE solutions :(

    Greetings
    ReCon

    • Check if your system supports VT-D first. Either in the hardware specs/BIOS or via “sudo xm dmesg | grep I/O” what should output something like “I/O virtualisation enabled”.
      If there is no such output, you’ll have no luck with PCI passthrough, sorry.

      Maybe I get the chance to test this on a ASUS Zenbook with a dedicated graphics card because this supports VT-D and be able to post the results here.

  7. I made it !!!

    Thanks a 1000 times for your how-to!!! It really helped.

    Note: I replaced my Nvidia Quadro 600 card for a Nvidia Quadro 2000 card which is advertised as “multi-OS” by Nvidia, and it worked.

    Since I messed up my Linux Mint 13 installation trying to compile xen with patches, I did a new install of everything. There were a number of things I had to adjust and add, like the pciback script to detach the secondary graphics card (Quadro 2000) and a USB host so I can use a USB KVM switch to switch my keyboard/mouse between Linux and Windows.

    I have now a running Windows 7 Ultimate 64 bit installation with VGA and PCI passthrough giving me a Windows performance index of 6.8 (lowest value = storage).

    Thanks for your help!!!

    • I’m so glad to hear you succeeded!

      Great that it’s working with NVIDIA graphic cards, too.

      Could you share your Adjustments with the PCI passthrough and explain them a little more so others may profit? Eventually I can add some facts above.

      Do you have real VGA passthrough working now or just passed the graphics card via PCI passthrough the way I did it? If you got VGA passthrough working, I am curious about details.

      • Well, “Nvidia cards” is too much a generalization. I replaced my Quadro 600 card for a Quadro 2000 card which Nvidia specifies as “Multi-OS”. Only these cards and drivers are supported officially by Nvidia for passthrough.

        Well, there seems to be some confusion about VGA passthrough and PCI passthrough. VGA passthrough, as I understand, is essentially PCI passthrough with some extra code that gets executed by Xen and the video card BIOS. Because of the reliance on the graphics card BIOS it’s only working with a few select graphics adapters.

        David Techer has produced a how-to and provided the necessary Xen patches for Nvidia cards. But these patches need to be compiled into the Xen hypervisor, together with the actual graphics card BIOS. All this is not necessary if you have a “Multi-OS” Nvidia graphics card, such as the Quadro 2000 (this is the cheapest in their lineup, and it’s not really cheap). AMD seems to have better support for VGA passthrough, though from what I read it once again depends on the model.

        With my hardware I didn’t have to do any compilations, patches, or VGA BIOS insertion.

        I’m working on a detailed how-to for others to follow. Here a short summary of the hardware and the process:

        Hardware:
        – i7 3930K CPU with C2 stepping (older 3930K processors have C1 stepping and are NOT supported)
        – Asus Sabertooth X79 motherboard with 1203 BIOS release and VT-d set to “enable”
        – Sapphire (AMD) 6450 graphics card as primary graphics adapter for Linux dom0
        – PNY Quadro 2000 (Nvidia) graphics adapter with “Multi-OS” as secondary graphics adapter for Windows 7 Pro 64 bit domU
        – Enough RAM (I got 32GB) and hard disk space for OSes and data
        – Linux Mint 13 Maya Mate 64bit OS for domU, with Xen hypervisor 4.1.2
        – Windows 7 Pro 64 bit for domU (Ultimate also works, so should other Win 7 versions)

        Steps:
        1. Boot LM13 from disk or better USB stick, install LVM2 package.
        2. Partition the hard drive(s): for MBR type partition tables (MUCH easier than EFI type booting) use: sda1 as ext2 for /boot; sda2 as LVM with ext4 formatted partitions for / and /home (and perhaps swap); an empty, unformatted LVM partition for guest OS (Windows domU).
        3. Install LM13 onto the pre-formatted partitions (the installer won’t recognize the partitions unless they have been previously formatted – see step 2).
        4. BEFORE reboot, chroot into the newly installed LM13 system, install LVM2 package, and run update-grup.
        5. Reboot into LM13, update packages, and install Xen hypervisor and some additional packages (I need to check exactly which one), including xen-pciback, all from the repository.
        6. Edit /etc/default/grub and comment out the “hidden menu” part, add XEN boot line parameters for IOMMU=1 and memory; run update-grub.
        7. Reboot into the Xen hypervisor/dom0.
        8. Create/copy pciback shell script into /etc/init.d, check lspci for PCI IDs of the VGA card as well as the USB host to pass through and create a pciback.cfg file in /etc/xen with the PCI IDs on separate lines, including the domain ID (e.g. 0000:02:00.0) – or do it as you wrote above and insert it into the script itself. Edit /etc/modules file and insert “xen-pciback passthrough=1″ – this is necessary to activate passthrough of PCI IDs to domU, instead of “vpci” which assigns a virtual PCI ID.
        9. Run update-rc.d, runlevels 2-5, with 19 21 to have the xen-pciback module loaded BEFORE xend starts running, and terminate it AFTER xend terminates.
        10. virt-manager and xen-tools use some different links than those used in LM13 and Ubuntu 12.04. Run as root:
        ln -s /usr/lib /usr/lib64
        ln -s /usr/lib/xen-default /usr/lib/xen
        ln -s /usr/share/qemu-linaro /usr/share/qemu

        11. Reboot again into the Xen dom0. Run xm pci-list-assignable-devices and check that all PCI IDs entered into the /etc/xen/pciback.cfg file are listed ! If they are, it means they are now removed from dom0 and can be assigned / passed through to the domU.
        12. Copy the Windows installation DVD as ISO image to hard disk – dd if=… of=… should work.
        13. Create /etc/xen/win7.cfg and adopt according to need. I’ll post mine later.

        More in my next post.

  8. Here is my /etc/xen/win7.cfg file for use with a Windows 7 Pro 64bit domU:

    $ cat /etc/xen/win7.cfg
    kernel = “/usr/lib/xen-default/boot/hvmloader”
    builder=’hvm’
    # Memory reserved for Windows domU, in my case 24GB (adjust to your
    # needs:
    memory = 24576
    # Name of the domU that will be created/used:
    name = “win7″
    vcpus=10 # I gave my Win 7 10 virtual CPUs, out of 12 (6-core CPU)
    pae=1 #may not be needed, but just in case
    acpi=1
    apic=1
    # Here my virtual network interfaces – see /etc/network/interfaces below:
    vif = [ ‘type=ioemu,mac=00:16:3e:68:07:07,bridge=xenbr0′ ]
    # I assigned a static MAC address, else it will be changed each time Windows
    # boots. The address should start with 00:16:3e., the rest is up to you.
    #
    # Specifying the disks and Windows ISO for installation, adjust to your
    # needs:
    disk = [ ‘phy:/dev/mapper/lm13-win7,hda,w’ , ‘file:/media/e119828d-6be6-46cb-83d8-a397dac5fd01/ISOs/Win7.iso,hdc:cdrom,r’ ]
    # More disks can be added later using this same method. The path to the
    # device or a file can be added. After the first comma is how the device will
    # appear. “hda” is the first, “hdb” the second etc. “hda” will appear as IDE,
    # sda will appear as SCSI or SATA. After the second comma “r” means read
    # only and “w” is for write.
    device_model = ‘/usr/lib/xen-default/bin/qemu-dm’
    boot=”dc”
    # The above should be changed once Windows is installed: boot=”c” to only
    # show the Windows file system, else it may try to boot from the ISO image.
    sdl=0
    opengl=1
    vnc=1
    vncpasswd=”
    stdvga=0
    # nographic=1 #!!! only uncomment this if you are using win8 or are trying
    # to get a Nvidia card to work. In my case – Nvidia Quadro 2000, this was not
    # needed.
    serial=’pty’
    tsc_mode=0
    viridian=1
    #soundhw=’all’ # I commented it out since it’s not relevant to me now.
    usb=1 # not sure this is needed, but it works.
    usbdevice=’tablet’ # is recommended, though not sure if needed
    gfx_passthru=0
    # Leaving this as 0 is how it works for me with my Quadro 2000 card.
    # You can change this later for iGPUs or nVdia if needed.
    # Try it with 0 first.
    pci=[ ’02:00.0′, ’02:00.1′ , ’00:1a.0′ ]
    # These values are the ones you found out by doing the lspci command earlier
    # I also passed through an entire USB controller for native support.
    # In Linux Mint you can use System Information/Input Devices GUI to find
    # out to which hub/host the keyboard/mouse is connected. I use a USB KVM
    # switch to connect my keyboard/mouse to two USB ports residing on
    # different hubs! One – 00:1a.0 – is then passed through to the domU.

  9. 14. My network configuration in /etc/network/interfaces:

    $ cat /etc/network/interfaces
    auto lo
    iface lo inet loopback

    auto xenbr0

    # I use a static IP for my NIC; you can also use DHCP:
    iface xenbr0 inet static
    bridge_ports eth0
    address 192.168.0.120
    broadcast 192.168.0.255
    netmask 255.255.255.0
    gateway 192.168.0.1
    bridge_stp off # disable Spanning Tree Protocol
    bridge_waitport 0 # no delay before a port becomes available
    bridge_fd 0 # no forwarding delay

    # There are no IP configurations for eth0, it’s all defined under xenbr0:
    auto eth0

  10. 14. This is actually a step that should be done earlier, when configuring the interfaces file just after installing the Xen hypervisor.

    Edit or create /etc/resolv.conf as follows:
    # Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
    # DO NOT EDIT THIS FILE BY HAND — YOUR CHANGES WILL BE OVERWRITTEN
    nameserver your.primary.DNS.server # e.g. 8.8.8.8
    nameserver your.secondary.DNS.server

    I removed the network-manager package as it gave me troubles. Not sure the above is the best approach, but it seems to work nicely. There are several Xen wikis and how-tos that explore this in detail.

    Here the /etc/modules file mentioned before:

    $ cat /etc/modules
    # /etc/modules: kernel modules to load at boot time.
    #
    # This file contains the names of kernel modules that should be loaded
    # at boot time, one per line. Lines beginning with “#” are ignored.

    lp
    rtc

    # For pciback to be loaded by kernel with PCI ID passthrough enabled, use:
    xen-pciback passthrough=1

    # Generated by sensors-detect on Fri Sep 7 15:04:55 2012
    # Chip drivers
    coretemp
    w83627ehf

    The above “xen-pciback…” entry is needed when xen-pciback is used as a module, that is if it’s not statically compiled into the kernel. Linux Mint 13 and Ubuntu 12.04 don’t have it compiled into the kernel, so its needed unless you like to compile your kernel.

    Here my /etc/init.d/pciback shell script (taken from above how-to and modified according to suggestions found elsewhere):

    $ cat /etc/init.d/pciback
    #!/bin/sh
    #
    ### BEGIN INIT INFO
    # Provides: pciback
    # Required-Start: $local_fs
    # Required-Stop: $local_fs
    # Should-Start:
    # Should-Stop:
    # X-Start-Before: xend
    # X-Stop-After:
    # Default-Start: 2 3 4 5
    # Default-Stop:
    ### END INIT INFO

    . /lib/lsb/init-functions
    scriptname=/etc/init.d/pciback
    configfile=/etc/xen/pciback.conf

    unbind() {
    # Unbind a PCI function from its driver as necessary
    [ ! -e /sys/bus/pci/devices/$1/driver/unbind ] || \
    echo -n $1 > /sys/bus/pci/devices/$1/driver/unbind
    }

    bind() {
    # Add a new slot to the PCI Backend’s list
    echo -n $1 > /sys/bus/pci/drivers/pciback/new_slot
    # Now that the backend is watching for the slot, bind to it
    echo -n $1 > /sys/bus/pci/drivers/pciback/bind
    }

    case $1 in

    start)
    cat $configfile | while read line;do
    echo $line | grep ^# >/dev/null 2>&1 && continue
    unbind $line
    bind $line
    done
    ;;

    stop|status|restart|force-reload)
    # As we don’t know which driver was bound before, there is not much we can do here
    ;;

    *)
    echo “Usage: $scriptname start” >&2
    exit 3
    ;;

    esac

    The above script needs to be “activated” using the upgrade-rd.c script as follows:

    $ sudo update-rc.d pciback defaults 21 19

    Should the above command give you errors, try update-rc.d pciback defaults (without the numbers). Actually update-rc.d should read the LSB header in the script to figure out when to run the script. Another way is using insserv, but it didn’t work in my case (complained about insserv not found, though it’s installed).

    Here is my /etc/xen/pciback.conf file – adjust to your needs (use lspci to get the right PCI IDs):

    $ cat /etc/xen/pciback.conf
    # This file contains the list of pci(e) devices the pciback driver should bind
    # to at system startup.
    # Please provide the devices in long BDF notation.
    # Example:
    # 0000:08:00.0
    # NO EMPTY LINES!
    #
    0000:02:00.0
    0000:02:00.1
    0000:00:1a.0

    And finally, though this is one of the first files to edit, the /etc/default/grub file:

    $ cat /etc/default/grub
    # If you change this file, run ‘update-grub’ afterwards to update
    # /boot/grub/grub.cfg.
    # For full documentation of the options in this file, see:
    # info -f grub -n ‘Simple configuration’

    GRUB_DEFAULT=0
    #GRUB_HIDDEN_TIMEOUT=0
    #GRUB_HIDDEN_TIMEOUT_QUIET=true
    GRUB_TIMEOUT=10
    GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
    GRUB_CMDLINE_LINUX_DEFAULT=”quiet splash”
    GRUB_CMDLINE_LINUX=””
    # I set the Linux Mint dom0 to use 2GB (2048M) of memory, see also
    # xend-config.sxp file in next comment.
    GRUB_CMDLINE_XEN=”iommu=1 dom0_mem=2048M”

  11. Here is the /etc/xen/xend-config.sxp file (shortened to remove the comments). Copy your original xend-config.sxp file for reference, it contains a lot of valuable information!!!

    $ cat /etc/xen/xend-config.sxp
    # -*- sh -*-

    #
    # Xend configuration file.
    #

    (xend-unix-server yes)
    # If you are using only one bridge, the vif-bridge script will discover that,
    # so there is no need to specify it explicitly. The default is to use
    # the bridge which is listed first in the output from brctl.
    #
    (vif-script vif-bridge)

    # This is a minimum both for auto-ballooning (as enabled by
    # enable-dom0-ballooning below) and for xm mem-set when applied to
    # dom0.
    (dom0-min-mem 196)

    # Whether to enable auto-ballooning of dom0 to allow domUs to be created.
    # If enable-dom0-ballooning = no, dom0 will never balloon out.
    (enable-dom0-ballooning yes)

    # 32-bit paravirtual domains can only consume physical
    # memory below 168GB. On systems with memory beyond that address,
    # they’ll be confined to memory below 128GB.
    # Using total_available_memory (in GB) to specify the amount of memory
    # reserved in the memory pool exclusively for 32-bit paravirtual domains.
    # Additionally you should use dom0_mem = as a parameter in
    # xen kernel to reserve the memory for 32-bit paravirtual domains, default
    # is “0” (0GB).
    (total_available_memory 0)

    # In SMP system, dom0 will use dom0-cpus # of CPUS
    # If dom0-cpus = 0, dom0 will take all cpus available
    (dom0-cpus 0)

    # Whether to enable core-dumps when domains crash.
    #(enable-dump no)

    # The tool used for initiating virtual TPM migration
    #(external-migration-tool ”)

    # The interface for VNC servers to listen on. Defaults
    # to 127.0.0.1 To restore old ‘listen everywhere’ behaviour
    # set this to 0.0.0.0
    #(vnc-listen ‘127.0.0.1’)

    # The default password for VNC console on HVM domain.
    # Empty string is no authentication.
    (vncpasswd ”)

    The rest of the settings are left to default as explained in the xend-config.sxp sample file (make a backup of it BEFORE tampering with it).

  12. 16. To actually install and boot up the Windows domU, I use the following commands in a terminal window:

    $ sudo xm new /etc/xen/win7.cfg
    $ sudo virt-manager

    The virt-manager should show the running dom0 and an entry for the win7 domU. Open the win7 domU and click the vnc window (terminal) icon, then click the “run” (>) button. If all goes well you will see remote desktop window with Windows installer booting.
    Select / answer the usual stuff for the Windows installer prompts. It will reboot a few times and eventually you get to remote desktop screen with Windows running. Open the Device Manager and select the graphics card with the yellow triangle (unless Windows has already found and installed a suitable driver). Let Windows search for a driver and do its job. In my case, it then automatically installed it and switched to the Nvidia Quadro card, switching my screen to the second DVI port and giving me a full screen regular Windows desktop (you may need to switch your screen to the second port manually by pressing a button on the screen). At this point the mouse/keyboard won’t work, so I switch the USB KVM to the second port. It takes a while for Windows to recognize the newly found hardware and install the drivers. Move the mouse to see when it responds to movement.

    Windows will do some updates and reboot perhaps several times. It should then always boot into the full desktop.

    If it doesn’t successfully reboot, shut it down using the virt-manager shutdown button. Wait until it says “shutdown” under win7. Exit virt-manager, then run:
    $ sudo xm delete win7

    Modify your /etc/xen/win7.cfg file to read:

    disk = [ ‘phy:/dev/mapper/lm13-win7,hda,w’ ]
    boot = “c”

    Then run again:
    $ sudo xm new /etc/xen/win7.cfg
    $ sudo virt-manager

    and just run the win7 domU with the start (>) button.

    I will hopefully be able to make some shell script to automate the process and run Windows at a mouse click.

  13. Some last remarks / caveats:

    I had problems booting my Linux dom0 today. It would freeze right after (graphical) login screen. No cursor movement, no remote access, nothing, only hard reset.

    I changed in the motherboard BIOS the following setting:
    Advanced menu -> USB configuration -> EHCI Hand-off = enabled
    That is supposed to solve USB issues with OSs that don’t support EHCI hand-off feature. It then booted without problem. Not sure this fixes it permanently and it may be a coincident. Need to check the log files for more.

    Another potential problem is the nouveau driver that was automatically installed with Linux Mint. It creates tons of error messages in the dmesg output as it has problems grabbing the Quadro 2000 card. Well, it actually shouldn’t even try it, as I pass this card to the win7 domU.

    I suspect that either the USB mouse/keyboard (or KVM switch which is a small USB host) or the nouveau driver are the reason for failed boots. Or perhaps the xen-pciback module gets loaded at too late a point and then somehow grabs the USB host where my keyboard/mouse are attached and passes it through to the non-existent (not yet running) win7 domU.

    Any ideas?

    P.S.: I’ll let you know if the BIOS adjustment provides a permanent solution.

    P.S.S.: There are some udev error messages that I like to fix, too. Perhaps via xend-config.sxp file?

    • In reply to my own questions above, here is how I solved it:

      1. blacklist the nouveau driver ! This solved the issue of failed boots (I booted/rebooted several times with no issues). I may remove nouveau later on as I don’t see any need for it.
      2. I removed the package bootchart – I installed it a few days ago to get a better analysis of the boot process and order of modules / drivers loaded. It showed me that xend is in fact started AFTER I loaded the xen-pciback module, as it should. However, with bootchart being started prior to init, it records every detail and takes a lot of resources. bootchart logs are compressed files that after opening are around 50-60MB large.
      3. Still got the udevd errors which I need to look into. But so far I can’t see any problems with the system, nor with Windows domU.

  14. Pingback: Xen PCI passthrough slides | kdj0c

  15. Pingback: Update to Ubuntu 12.10 | kdj0c

Leave a Reply