Friday, June 15, 2012

gaming on gentoo/linux via xen


xen - a hypervisor who is one of many hardware virtualization techniques which allows running concurrently guests on a host system. One of the special key features of xen is vga passthrough which allows a guest to have full access to a pci device.
When i bought my new PC i wanted to use that feature because i missed gaming on the PC and since i don't wanted to reboot every time i decided to setup a windows 7 guest with a passthroughed graphic card.
In this post i'll gonna explain my setup and show what i have done to get it work.

My problem was easy to describe. I wanted gaming. Wine was an opinion, but i don't wanted to "frikle" around to get games working and multiboot was a no-go. Even though boot times are already pretty fast i wanted to have linux running all the time. This brought me to xen...

linux and windows side by side

1) Setup

First, my hardware setup and how it should work.
I have a an 8core amd processor with 16GB ram and two graphic cards. 4cores, 8GB ram and the high end graphic card is solely for the virtualized windows. That's enough power to run most new games without problems. (aside of linux)
Both graphic cards are connected to my 30" monitor and with the input key on the monitor i can switch between windows and linux.
The linux graphic card has two more 22" screens on the left and right of the 30".

2) Installation

2.1 System:
First i made a typical gentoo installation. It's ~amd64 keyworded, means it's more or less bloody edge. I even have a few 9999 packages installed like mesa, libdrm, xorg-server, etc but that shouldn't have any influence on xen.

make.conf:
 CFLAGS="-O2 -pipe -fomit-frame-pointer -mcx16 -msahf -maes -mpclmul -mpopcnt -mabm -mlwp -mavx -march=native"  
 CXXFLAGS="${CFLAGS}"  
 CHOST="x86_64-pc-linux-gnu"  
 MAKEOPTS="-j14"  
 ACCEPT_KEYWORDS="~amd64"  

With gcc-4.6 i'll add "-march=bdver1 -mtune=bdver1" to CFLAGS for enabling all possible features of the new bulldozer cpu.

Another change was to create a second runlevel especially for xen. I already bloged about it so you can read the small howto here: Link

2.2 Network:
I created a bridge so that xen can create tap devices for the virtual guest. A good howto for creating bridges can be found here: Link

2.3 Xen:
Really important is to have a multilib profile on gentoo, otherwise you couldn't set the "hvm" flag on xen-tools which is required for the amd-v processor feature. (i had to install gentoo a second time because of this)
The installation of xen was really smooth. I enabled following flags, where at least the "hvm" flag is really important.

xen flags:
 pp-emulation/xen-tools hvm qemu screen  

I also tried out xen from git because it has lots of vga passthrough patches, but since i couldn't start any virtual machine with the git version, i switched back to the latest version in portage. The git version of xen can be found in the virtualization overlay.

2.4 Kernel:
On the kernel side i compiled almost everything which has to do with xen directly into the kernel. I choose to use the latest git-sources so that i have all the latest features. I'll provide my configuration here for those who are interested: Download

2.5 Grub:
Grub also needed a new entry for xen which looks like this:
 title Gentoo Linux XEN  
 root (hd0,0)  
 kernel /boot/xen.gz dom0_mem=8192M,max:8192M iommu=1 xsave=1 dom0_max_vcpus=4 dom0_vcpus_pin                                                                          
 module /boot/gentoo-3.4.0-rc6 root=/dev/md3 softlevel=xen xen-pciback.permissive xen-pciback.hide=(01:00.0)(01:00.1)(00:12.0)(00:12.2) pci=resource_aligment=01:00.0,01:00.1,00:12.0,00:12.2  

Below the descriptions about this configuration. Dom0 is the host system, all systems which are virtualized are domU's

dom0_mem=8192M = the host system get's 8GB Ram...
,max:8192M = and can't have more than 8GB
iommu=1 = turn iommu on
xsave=1 = don't know exactly what it does, but it was need to prevent a bug with avx, see here: LInk
dom0_max_vcpus=4 = the host system get's 4 cpu's
dom0_vcpus_pin = it pin's the first 4 core's solely for the host
softlevel=xen = see here
xen-pciback.permissive xen-pciback.hide=(01:00.0)(01:00.1)(00:12.0)(00:12.2) pci=resource_aligment=01:00.0,01:00.1,00:12.0,00:12.2 = this is actually for pci(vga) passthrough. The bus number of you're graphic card and/or your usb bus from lspci can be found with lspci. It hide's the first pci device - the hd7970 (01:00.0 & 01:00.1) and one usb bus (00:12.0 & 00:12.2) which i also passthroughed to the guest. Note, you always have to hide everything on a single bus e.g. 01:00.0 & 01:00.1 and not just 01:00.0

3) Windows 7

3.1 Installation:
The installation of Windows 7 went really smooth. First i've created an 30GB empty image with dd:
 dd if=/dev/zero of=win7_64 bs=1M count=30000  

Then i've wrote a config file for xen:
 name='win7x64'  
 builder='/usr/lib/xen/boot/hvmloader'  
 builder='hvm'  
 vcpus='4'  
 cpus='4-7'  
 memory='6144'  
 on_poweroff='destroy'  
 on_reboot='restart'  
 on_crash='destroy'  
 disk=['file:/home/michael/xen/win7x64s.img,hda,w','file:/home/michael/xen/win7.iso,hdc:cdrom,r']  
 vif=['bridge=br0,type=ioemu']  
 boot='c'  
 acpi=1  
 apic=1  
 viridian=1  
 stdvga=0  
 vnc=1  
 vnclisten='0.0.0.0'  
 vncdisplay=0  
 vncunused=1  
 vncpasswd=""  
 sdl=0  
 pci=['01:00.0','01:00.1','00:12.0','00:12.2']   
Download this config: Download

The last line is important, because this one says which pci buses are gonna be passthroughed to the guest. I also enabled vnc, so that i can connect to the system with krdc which was important while installing windows.
"disk" is the option where you point to the image and also to the iso of the windows7 dvd. A better explanation of all options can be found here: Link and as PDF

Note:
It's important not to turn on gfx_passthru, because with xen-4.1.2 vga passthrough isn't supported at boot time on ati devices. First you have to install windows and than the driver for you're gpu. Afterwards windows want to restart the system to enable the new gpu - do that and you'll have you graphical output on your second gpu :)
As long as you don't want to try out xen-unstable that's the way to go, which means as for now you don't need the gfx_passthru option.

To start xen you just have to type following into a console (as root):
 xl create /path/to/the/win7-config.cfg  

Next you start your favorite vnc viewer and connect to localhost and install windows. Afterwards you'll have to download the windows ati driver and install them. But be sure to install just the drivers without the ati ccc. It's known that the ati ccc makes troubles with xen. Don't install it.
I think it's the best way is to extract them and let windows install the drivers over the device manager.

3.2 Audio:
I've choose to create a Windows 7 64Bit Version. Actually, when i started to though about xen my biggest worries where everywhere except audio. I actually though it's pretty easy to get audio to work, but actually it's almost impossible.
Why? Well, xen doesn't have any emulated audio cards which work with windows7 64bit. I really searched long for a solution and the final one is more or less a workaround.
I bought a new external USB audio card for about 20€ and connected it on the windows usb bus where windows immediately installed the driver.
The usb audio card has one output and one input. From the output of the usb audio card i go into the input of my onboard audio card on linux. Now i can easy turn on/turn off the sound in windows over linux. Crazy - isn't it? :)
And the cool thing is i can run music on linux while playing on windows at the same time since the virtualized windows is just one more input for linux.

This is the audio card which i bought:
USB Audio Card
Product page of the sound card: Link

3.3 Input:
Input was also a important topic. With pci passthrough input devices are also handled different. The problem is that the monitor output is not on the host system thus you also can't move with the mouse from the hostsystem like in qemu or virtualbox.
That's why, besides the graphic card, i also passthrough an complete usb bus. So i had 4 usb slot solely for windows. I already needed one usb slot for the external audio card. Now i needed two other for the mouse and keyboard.
Furthermore i don't wanted to have 2 mice and 2 keyboard for technically one system and so i searched for a switcher like a kvm switch but just for usb. I found this one. It works just for one device so i had to buy it two times.
Delock USB Switcher
Pretty smart isn't it :) It works, but it's not perfect. Here is the product page of this device: Link

3.4 Harddisks:
Besides the 30GB harddrive for the windows7 system i also created a 300GB harddrive just for data. There i store all my games. In case something goes wrong with windows i still have all my games one separate (virtual) partition.

4) Problems

So far everything works really flawless, but there are still some problems.

  • I wouldn't count the audio devices as problem, because it works really good but it's sad that xen still can't emulate any modern audio cards.
  • It was also a pain for me when i found out that xen supports passthrough first after you installed windows and installed the ati driver. I tried alot after i found that out. Anyway, this will come with xen-4.2.
  • Another problem right now is actually the usb switcher - and that one is also just a minor one.It works, but mostly just from windows to linux. Linux actually never want to release the keyboard/mouse and so i always have to unplug the devices first. It's annoying, but its ok :)
  • My biggest problem right now is that once you shut down the virtualized windows you cannot start windows again. I have to restart the whole linux system too, otherwise it won't work. A bit annoying but i hope that will be fixed with xen 4.2.
Lastly i made a short demonstration video showing UT3 with max details on my 30" screen. I know, it's not the newest game out there, but i think it still has a really good graphic and actually i don't have anything better. (newer yes, but i guess diablo 3 isn't such a pretty game :) )


Btw, sorry for the bad quality - it's my first video on youtube.
That's all for now. Next step will be xen unstable, kernel 3.5 with latest gcc xD