Sunday, March 26, 2017

news on my qemu-kvm init script for gentoo

It's been a while since i mentioned my kvm init script in any of my blog post. However, it's still alive and i added a lot of new features over the time. Anyone who is still interested can download it's source on github [1] - it's been there for quite some time.

So what changed since last time i wrote about it? Well, lets see:

  • virgl support (alias -gl)
  • 9p support
  • sdl output support
  • parallel spice and vnc support (enabling spice and vnc in parallel)
  • vnc with password support
  • multiple network cards with support for different bridge devices
  • usb redirect support
  • reset virtual machines (via /etc/init.d/kvm.VM reset)
  • reboot via qemu guest agent (via /etc/init.d/kvm.VM reboot)
The init script in action

Besides that, almost everything in the configuration file changed in order to support all the new features and make it more readable. Another plus it that the init script now should be posix compliant. Furthermore i tried to keep it's dependencies low. Apart from sys-apps/coreutils everything you should need is:

  • app-emulation/qemu[pyhton]
  • net-analyzer/netcat6
  • sys-apps/iproute2
  • sys-process/lsof

I'm working with this script on a daily base and i think it's already very convenient. I also think it still has room for improvements and i'm going to continue adding features as i need them. However, please let me know if you're missing some features. I would like to know what else i could implement.

[1] https://github.com/mm1ke/qemu-init

Monday, February 6, 2017

games depending on pulseaudio

I haven't posted something for quite some time. However, since every now and then I have some idea which i would like to share, I forced myself to start writing again. Hope i can keep up with new posts as new ideas come up in my mind.

Today i would like to share something about steam games which need pulseaudio. I think some of you might already experienced some issues with games when you haven't installed media-sound/pulseaudio. Most notably was probably Tomb Raider.
In that game you haven't had any audio as long as pulseaudio wasn't installed. Since there was quite some noise on the steam forums, they fixed the issue and now it works well even without pulseaudio.
However, Tomb Raider isn't the only game with pulseaudio issues. Another game which made me quite some headaches was Audiosurf 2. The Problem: Besides no audio i couldn't even play the game and it showed me that i was running only the demo. That means, for me, it wasn't clear that pulseaudio was the problem. A long time after writing a post at the steam forums i got some hint's that pulseaudio might be the problem.

Anyway, that time i already fixed the issue differently.
Since Tomb Raider came out I've searched for a way to play the game without pulseaudio. Sadly i couldn't find anything, but found a way were pulseaudio is only a unintrusive pipe to alsa.
I don't know were exactly I read it, but i found a very nice wiki entry on arch [1] , which also worked as expected on gentoo:

First of all i had to install pulseaudio. For that package is important NOT to install the alsa-plugins so first i had to remove the alsa-plugins use flag:

echo "media-sound/pulseaudio -alsa-plugin" >> /etc/portage/package.use

After emerging i it was important to make some configuration changes for pulseaudio:
/etc/pulse/daemon.conf: (everything else in this file is commented out)
#/etc/pulse/daemon.conf
exit-idle-time = 0 # Exit as soon as unneeded
flat-volumes = yes # Prevent messing with the master volume

/etc/pulse/client.conf (everything else in this file is commented out)
#/etc/pulse/client.conf
autospawn = yes

/etc/pulse/default.pa (this file was completely replaced by following)
#/etc/pulse/default.pa

.fail
    # Set tsched=0 here if you experience glitchy playback. This will
    # revert back to interrupt-based scheduling and should fix it.
    #
    # Replace the device= part if you want pulse to use a specific device
    # such as "dmix" and "dsnoop" so it doesn't lock an hw: device.
    
    # INPUT/RECORD
    load-module module-alsa-source device="default" tsched=1
    
    # OUTPUT/PLAYBACK
    load-module module-alsa-sink device="default" tsched=1 
    
    # Accept clients -- very important
    load-module module-native-protocol-unix

.nofail
.ifexists module-x11-publish.so
    # Publish to X11 so the clients know how to connect to Pulse. Will
    # clear itself on unload.
    load-module module-x11-publish
.endif

With that changes it was possible to play Tomb Raider and also didn't had any problems anymore with Audiosurf 2. While of those games are already quite old, someone might find this information useful for other games.

[1] https://wiki.archlinux.org/index.php/PulseAudio/Examples#PulseAudio_as_a_minimal_unintrusive_dumb_pipe_to_ALSA

Tuesday, June 30, 2015

removing old backups

Ever had to think about how to remove old backups? Like you want to keep the last 30 backups? Whenever i had to think about a solution to this i though about something with "find -mtime". However, this only works when backups were made constantly on a daily base.

But what happens if a backup fails or an external server doesn't have a connection to the storage? In my case my laptop only sporadically creates backups. If my laptop would be turned off for 30 days all of my previous backups would be deleted with "find -mtime".

Until now i had a huge script which checks for such cases. Just stupid...

Today i found THE solution!
A really easy and nice one-liner to always keep the last 30 backups. It's just so nice :D

Attention: Don't simple copy/paste this one-liner - it can remove files without asking!

find /backup/folder -type f -printf '%T@ %p\n' | sort -k1 -n | head -n-30 | cut -d' ' -f2 | xargs -r rm

I think i don't really have to explain it. It keeps the last 30 backups - doesn't matter how old they are, but they are always the newest one. In case you have multiple backups make sure to keep them in separated directories or filter them with "find -name "*keyword*"". And before using this one-liner i strongly suggest removing the last part (xargs -r rm) to see what would be removed.

Hope someone can find it useful. I've searched hours to find something like this and never found anything. (probably because i searched with the wrong keywords...)

Monday, June 1, 2015

testing is fun (binpkg-multi-instance)

Since version 2.2.19 (now 2.2.20) portage implemented a feature called binpkg-multi-instance. This is a feature which i was looking for quite some time. In the last days i had some time and i decided to test it.
The feature itself brings the ability to portage to keep multiple versions (with different use settings) of a single package version.
Until now, if you created a binary package, portage could only keep exactly one binary-version of any package. If you build the package again with different use-settings and created a binary package the version prior would had gone.

Now this is probably not something many people were looking for. I was one of those who were really exited about it. When the feature hit git i was already tempted to test it directly from git head.

So why is that so exciting for me? Well, because some time ago i set up a nice test system where this feature helps _alot_ keeping compile times at a minimum.

Background:

The idea was simple. How to test many different setups and keep compile times at a minimum?
I wanted a base system which i could clone anytime i want so that i could install and test various package combination's and use already compiled packages as much as possible. That being said, a virtual machine with snapshots did come into mind. However, i had a dedicated hardware which had nothing to-do and thus there was no need for virtualization. Another candidate was btrfs with it's snapshot features. The problem here: What if i want to test another filesystem? ;)

Logically i decided to go with lvm.

The boot partition is shared with every system. Every other partition is on lvm. Every root partition is unique, only /home, /usr/portage and /usr/src are on a separate lvm partitions as those can be shared as well.
First i've created a "base" gentoo system. Basically a stage3 system with some additional programs and a few important settings.
EMERGE_DEFAULT_OPTS is one of the most important in this case. In my case it looks like following:

1
EMERGE_DEFAULT_OPTS="--binpkg-changed-deps=y --binpkg-respect-use=y --buildpkg --buildpkg-exclude \"virtual/* sys-kernel/*-sources\" -k --jobs=3"

It tells portage to always use, if possible, binary packages and, except for kernel-sources and virtual packages, to always create binary packages. Since this setting is my base system it's in every clone of it. (as long as i don't change anything by hand)

And that's were binpkg-multi-instance comes into mind. Since every system access the same binary package store, but every system might have different use setting for a particular package, every package now only has to build once in any case!

Compiling is really funny right now, cause it looks quite often similar like here:

Conclusion:

Sure, the whole setup is of course a bit more complex and while this setup works really great there are a few things to mentioned. For example, the kernel(s) needs a few features in every system (like lvm snapshot, openrc and systemd - if i want to test both, which i do). Also since home is shared with every system, testing various window managers (like kde,gnome,xlqt,...) could mess up their configurations. Also having different arches (x86 and amd64) need adjustments to the base configuration. (but it's working too!)

Besides that i've also wrote a small script which does most of the work. It clones and installs (grub) any system at any moment even with a different file-systems if desired. (plus it can also encrypt a cloned system).
For example, basically all i have to-do is:
1
./sysed -q
This clones the actual running system with the actual file-system and size and creates an grub entry which is called "${lvm_name}_testing".
The script can also backup, restore, delete and edit my lvm systems.

I'm using this script quite often as it's really simple cloning a whole system in about ~2 minutes. So far i already have 14 amd64 and 2 x86 systems. Below a list of my systems (from lvs).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
  gentoo_amd64_acp               vg0  -wi-a----- 10.00g
  gentoo_amd64_base              vg0  -wi-ao---- 10.00g
  gentoo_amd64_base_selinux      vg0  -wi-a----- 10.00g
  gentoo_amd64_base_systemd      vg0  -wi-a----- 10.00g
  gentoo_amd64_cinnamon          vg0  -wi-a----- 10.00g
  gentoo_amd64_enlightenment     vg0  -wi-a----- 10.00g
  gentoo_amd64_gnome             vg0  -wi-a----- 10.00g
  gentoo_amd64_kde               vg0  -wi-a----- 10.00g
  gentoo_amd64_kde_testing       vg0  -wi-a----- 10.00g
  gentoo_amd64_lxqt              vg0  -wi-a----- 10.00g
  gentoo_amd64_mate              vg0  -wi-a----- 10.00g
  gentoo_amd64_qtile_systemd     vg0  -wi-a----- 10.00g
  gentoo_amd64_sec               vg0  -wi-a----- 10.00g
  gentoo_amd64_secure            vg0  -wi-a----- 10.00g
  gentoo_x86_base                vg0  -wi-a----- 10.00g
  gentoo_x86_kde                 vg0  -wi-a----- 10.00g

binpkg-multi-instance had an big effect here, especially when trying things like abi_x86_32 or selinux. From now on i won't have to compile any package a second time anymore as long as i already build it once!

Big thx to the gentoo portage team!

Thursday, May 28, 2015

less portage rsync output

Ever thought how to silence rsync when doing emerge --sync (or eix-sync). Sure, it's nice when we get lots of information. But it's like with compiling packages - The first few years it's amazing looking at the terminal while it's compiling the latest stuff. However, after a while these things become a bit boring.
While we have --quite-build for emerge, rsync per default outputs every single file which gets transferred and deleted. Luckily, recent versions of rsync, which also went already stable, support new ways of progress output and since i use them already on other scripts i decided to modify my portage rsync settings a bit:

1
PORTAGE_RSYNC_EXTRA_OPTS="--info=progress2,name0,del0"

The output looks similar like that:


Neat, isn't it?
BTW, in order this works correctly - the remote rsync server need to run a recent version of rsync as well.

Monday, August 25, 2014

flashing android mobiles on gentoo

This is just a quick tip in case you ever want to flash a mobile phone on gentoo.

If you look at the cyanogenmod howto [1] (in my case for a nexus s) you'll see that you need the tools "adb" and "fastboot" which usually comes with the android sdk. Naturally the howto suggests you to install this sdk, which isn't even available on gentoo.
However if you don't want java and all it's other dependencies on your computer (which is required for the sdk) there is package which installs only those two needed tools. It's called dev-util/android-tools - and it's in portage :)

This is all you need:
* dev-util/android-tools
     Available versions:  (~)0_p20130123
     Homepage:            https://android.googlesource.com/platform/system/core.git/
     Description:         Android platform tools (adb and fastboot)

[1] http://wiki.cyanogenmod.org/w/Install_CM_for_crespo

Sunday, August 10, 2014

jumping directly into found results in menuconfig

For those who still use menuconfig for configuring their kernel - there's a neat trick which let you jump directly into a found result.

For example you would like to add a new driver. Usually you go into menuconfig and start searching for it with the "/" shortcut. What you probably not know, after you found your module - like you searched for the "NetXen Multi port Gigabit Ehernet NIC" with just searching for "xen" - you can go directly to the particular config via it's number shortcut:
Search result for "xen"












Notice this line:


The "(5)" is the shortcut. Just press the number 5 on your keyboard and you'll jump directly into the QLogic devices config.
For every found entry there is a number shortcut which let you directly jump into the given config. If you go back with esc-esc you also go back to the search result.

I think not many people know this trick and i hope someone can use it for further kernel builds ;)