Tuesday, December 20, 2011

oxygen style for gtk+ applications

GTK+ applications usually dosen't look good under kde. Ever since i'm using kde i always tried to avoid using gtk+ applications under kde. That worked out quite good because i always found a qt application which fullfill my needs. There are just a few applications where kde/qt doesn't have a real competor to the gtk+ version.
A big problem was always the webbrowser. Now we have rekonq which is really good but it's still quite buggy. Another programm is gimp. I couldn't find a real competitor on the qt side (krita is basically a different audience).
Another way is to live with gtk+ applications is to use a widget style which works for both gtk+ and kde/qt, but i actually never liked the qtcurve widget style at all.
Recently i found another widget style which is called oxygen-gtk. This is the official port of kde's widget style to gtk+.
Well this one looks really good and makes my desktop much more "pure kde" :)
In gentoo's case you just have to:

emerge -av x11-themes/oxygen-gtk


Afterwards go into System-Settings --> Application Appearance --> Gtk Config. There you set the Theme to "oxygen-gtk".
The Result looks something like this (in the picture you see Gimp)

Wednesday, December 14, 2011

howto mount qcow2 images

Sometimes you need a way to access a virtual machine without starting it. It might dosn't work anymore and there are still some important files on it or you are simple just to lazy to start, login and change/look whatever you want to change/look for.
In case of lvm it easy: There are "real" partitions on the volume so its dead simple to mount and access the guest harddisk. It's also simple with raw files which works with loop devices thus with losetup.
Another story are qcow2 files.
For them you need qemu-nbd which comes with app-emulation/qemu-kvm and a kernel modul called nbd which you will find under: "Device Drivers --> Block devices --> Network block device support". If you didn't compile it yet (as modul) compile it now.
Next you have to load the modul with following parameters:

modprobe nbd max_part=8

You have to set the max_part because the default is 0 and usually you have more than 1 partition on a guest system. After that you can mount the image with:

qemu-nbd -c /dev/nbd0 guest.img
mount /dev/nbd0p1 /mnt/guest

That's all. With qemu-nbd -d you can disconnect the image finally, though you have to unmount it first.

Sunday, December 11, 2011

btrfs and virtualization

Btrfs - it 's the new and feature-rich filesystem which should be eventually the next standard fs in the linux world. So long it's still in development, but it's already in the linux kernel. That's why i also gave it a try, especially for virtualization.

The feature which i'm most locking forward is snapshots. Making backups of running guests is always a cool thing :) After months of running guests on btrfs i can say I'm both exited and also a bit disappointed of btrfs.

First of all: The default settings in qemu makes the guest extremely slow. I tell you why. Usually if you start a guest, his drive cache settings are set to "writethrough". This setting controls how the host cache is used to access block data. You can choose between between "none", "writeback", "unsafe" and "writethrough" with the last one being the default.
On btrfs this makes the guest extremly slow. I made some benchmarks on lvm, ext4 and btrfs comparing "writeback", "writethrough" and "none". It's a simple comparison of how long it takes to compile dev-vcs/git with the different settings on the different filesystems:


lvm:
  writeback  merge time: 1 minute and 39 seconds. 
  writethrough merge time: 1 minute and 50 seconds.
  none   merge time: 1 minute and 44 seconds.


ext4:
  writeback  merge time: 1 minute and 39 seconds.
  writethrough merge time: 1 minute and 55 seconds.
  none    merge time: 1 minute and 46 seconds.


btrfs:
  writeback  merge time: 1 minute and 41 seconds. 
  writethrough merge time: 2 minutes and 37 seconds.
  none    merge time: 1 minute and 48 seconds.


As you can see on btrfs the compilation almost doubles the compilation time with "writethrough". Luckily, that problem is already known and at least for me it's a reason not using btrfs for virtualization.

Another downside is that btrfs still doesn't has a fsck like tool. I've used btrfs for some months now and one of my windows7 guests got corrupted. Gladly it was just a test-system. Anyway i had to delete it and made a new installation.
On the other side, the snapshot feature is really nice. I've wrote a easy backup script for the image files which worked really good.

Right now i moved all my guests to lvm (again) because i think it has still the best performance, even though i need much more space. But at least it's also possible to create live backups.

Saturday, December 10, 2011

alix - logging

As i already mentioned in a previous post, i use my alix as a firewall. Usually a firewall's job is also to log attacks against the firewall.
But logging means writing to a log-file and since the hard-disk is a compact-flash card i tried to avoid that. But syslog-ng is quite powerful and it has the feature to log to another host. That's why i set up my server to receive log's from other host. Right now everything is without encryption, but for example with net-misc/stunnel you could even encrypt your live logging. The setup looks like this right now:


On the firewall i have this in syslog-ng.conf:

source s_local { internal(); unix-stream("/dev/log"); file("/proc/kmsg" log_prefix("kernel: ")); };
destination d_loghost {tcp("$serverip" port(514));};
log { source(s_local); destination(d_loghost); };

Every other log { } event is commented out on the firewall, so that syslog-ng doesn't write anything on the compact-flash card. On the server i had to change the config also a bit. First, it has to accept connections from outside which looks like this:

source src {
    unix-stream("/dev/log" max-connections(256));
    ...
};

Secondly, I've created some filters especially for the firewall rules:

filter f_iptables_drop { match("^iptables: INPUT DROPPED" value("MESSAGE")); };
filter f_iptables_access { match("^iptables: ACCESS*" value("MESSAGE")); };

The first filter, logs all the failed connection against a closed ports. The second one logs all the accepted connections against an open port. To separate them in different files i also have this in my config:

destination d_firewall_drop { file("/var/log/clients/$HOST/$HOST-firewall-drop"); };
destination d_firewall_access { file("/var/log/clients/$HOST/$HOST-firewall-access"); };

The rest goes logged into:

destination d_clients { file("/var/log/clients/$HOST/$FACILITY"); };

To be able to filter the log in such way you also have to mark the dropped and accepted packages so that syslog-ng can differ them. To do that you have to add the following to your firewall:

iptables -A INPUT -m limit --limit 15/minute -j LOG --log-level 7 --log-prefix "iptables: INPUT DROPPED: "
iptables -t nat -A PREROUTING -i ${NET_ETH} -p ${PROTO} -d ${NET_IP} --dport ${PORT} -m limit --limit 15/minute -j LOG --log-level 7 --log-prefix "iptables: ACCESS ${PORT} "


That configuration works well for already some time now. Maybe someone finds the information useful.

Wednesday, November 9, 2011

qemu-kvm: spice

Since i'm playing around with virtualization there was always the problem of accessing my virtual machines. Usually the guests always running on my server, means i have to access the screen over a remote control protocol. Actually there are a few techniques out there.
  • vnc - very popular and stable, but sadly quite slow
  • rdp - very fast and most importantly it has copy/paste support, but limited to windows
  • xdmcp - limited to X and needs lots of bandwith
And there is this new and shiny spice. Well, actually it's not shiny. Since it's quite new, there is not even a good client. They are lot's of other pros and cons about the few above, but actually i want to talk about spice.
So, what's really good about spice? Well it's fast and it supports copy/paste (well, it should, i couldn't get it to work so far).

Another good thing is that they are also drivers for windows, means it should work on windows too. I didn't try it out on windows and won't do that for quite some time since the rdp clients are pretty stable on linux.

Anyway, on linux it was worth trying it out. It's really fast and it's also very stable even though the client is really basic. I just hope i get copy/paste working in the near future. Technically spice has much more pros. Its written from scratch and should be the future technique for interacting with virtualized systems especially based on qemu.

While i played around with spice i also decided to update my gentoo init scirpt a bit. Now it has "basic" spice support (you can enable/disable it) and i also fixed some bugs and improved the default configuration.
Just comment out VM_HAVE_SPICE="false", but be warned: vnc will be automatically turned off.
Also, to have spice support in qemu, you have to add the "spice" USE-Flag for qemu-kvm.
As client i suggest net-misc/spice-gtk. Well it's the only one i found.

Here is the download link for the scirpt: Link
And here is the link to my previous post about the script with a small howto: Link

Monday, October 3, 2011

pigz: the parallel implementation of gzip

Backuping is a very important todo nowadays. Even i backup constantly my whole systems. Mostly with lvm + tar.
Recently i've started to play around with btrfs and also with those snapshot features. Since on this partition are just virtual machines and since btrfs is still under development i regularly make a backup of those images. This leads me to gzip. My usual solution for copying and instantly compressing images to another partition is done by:

dd if=/path/to/image | gzip -c > /path/to/backupfolder

Now the problem is: gzip is single threaded, which means it's slow, especially on a multicore system, like mine which has 8 cores.
Asking google i found a nice replacement for gzip, called pigz. The really cool thing is, it's syntax is exactly the same like for gzip which means i don't had to change my code, i just had to change the command.

The improvement is enormous: (a image with 2,9 GB, simple stopped with "time")
gzip:
real    3m50.361s
user    3m43.210s
sys     0m10.480s

pigz:
real    0m37.080s
user    4m2.370s
sys     0m16.070s

Really nice, isn't it? Now in combination with my backup scripts for my gentoo systems, it's a really neat upgrade.

Monday, September 26, 2011

qemu-kvm: balloon driver

Qemu-kvm has a really neat feature called memory ballooning. This feature allows you to change the memory for your virtual guest doing runtime. Usually this feature is enabled by default, but the virtual guest also has to support this (By default my gentoo initi-script also turns it on). To make it work on the guest side you only have to enable the following feature in the kernel config (guest kernel):

CONFIG_VIRTIO_BALLOON=y (Virtualization -> Virtio balloon driver)


There are even drivers for Windows which are available here: Link
After turning on you can connect to the qemu monitor via netcat or telnet. Too see the actual memory of the guest you only have to type:

(qemu) info balloon


The syntax for increasing or decreasing the memory is:

(qemu) balloon 1024


This command will change the memory allocation to the specified amount in MB. Looking at the above example it would be 1024MB.

Tuesday, August 23, 2011

gentoo qemu/kvm init script

Since i have a really powerfull server i also play alot around with virtualization. In particular with qemu/kvm. Right now i have 2 guest running all the time. One is a hardened gentoo with a ftp server on it, the other one is a windows xp system. Already when i bought my server i decided to write a start/stop script for my qemu/kvm guests, which i want to share now with the Internet.
I already spend lots of time in this scirpt and i think it's quite useable, though it's far away from perfect, but i still improve it. It's current features are:
  • Start a vm based on their name from the init-scirpt.
    I gonna explain that more detailed:
    When i started to write this script, i wanted to have something similar like the net.lo scirpt in gentoo. For every network device in gentoo will be a new link created to net.lo. The configuration for every device is also in one file. While making a link for every vm is a great deal, having all the configuration in one file isn't a good idea (the config file is quite long), means for every vm exists a seperate config file. Now, the default scirpt is called kvm.init, a start script for a vm is then a link to kvm.init and is, for example, called kvm.winxp. Now, it's important to find the image of the vm.
    Either their is the complete path to the image in the config (/etc/conf.d/kvm.winxp) or there is just the directory in the config means that the script has to find out the image name based on the name of the init scirpt.
    Example: kvm.winxp = it would look for a image called winxp, winxp.img or winxp.qcow2. (no matter which ending it has). Does it find more possible images it wouldn't start anything and gives back an error.
  • Every guest gets an tap device for the network (as long as network is wanted), means every guest has a full featured network device with acces to the local network. The scirpt generates for every guest a tap device on the host system. A bridge has to be setup before. On shutdown, the scirpt deletes the tap device.
  • The script checks other vms for same mac-address, vnc-addresses and ports or images which runs already under a different script. Usually the script adds these options by itself correctly.
  • For many options the scirpt checks for it's correctness.
  • A vm will be shutdown via nc through qemu monitor, giving the guest 80 secounds time to shutdown. That should work at least on windows and linux (as long acpid is installed and runs)
  • Qemu can be run as an different user, even the tap devices can be created under a different user
  • The scirpt supports features like viritio-console, virtio-balloon, virtio-net, virtio-blk or vhost for better/faster virtualization (guest has to support this)
  • Well documented configuration file. I tried to make it as easy as possible.

What is needed for this scirpt:
  • Since it generates a tap device for every guest, you must have a bridge configured!
  • The srcipt itself needs following commands: sleep, rm, ps, [e]grep, brctl, ip, nc, ls, head and wc, thus needs following packages installed: 
    • sys-apps/coreutils
    • sys-process/procps
    • sys-apps/grep
    • sys-apps/iproute2
    • net-misc/bridge-utils
    • net-analyzer/netcat6
  • a processer with vt-x support
  • a kernel with virtualization enabled

How does it work:
  • First of all, download my kvm start-stop script from here: Link
  • Copy kvm.init into /etc/init.d/
  • Make a new link for every guest with: "ln -s kvm.init kvm.windows7"
  • Copy kvm.config into /etc/conf.d/
  • Rename kvm.config into kvm.windows7 (like the init script)
  • For every new guest you have to copy the config again.

Be warned, i never tried this init-scirpt on other machines, so there is no guarantee that it works. Please make sure you have all programms installed which are needed and also have the config file edited to your needs. Especially the path to the image and the bridge device has to be set. I also strongly suggest to create an extra directory for the pid files. (it's set to /var/run/kvm per default)
Have fun with the script. Suggestions, bugs and improvements are welcome :)

The init-script: Link

    Sunday, August 21, 2011

    python: reverse numbers

    Recently i've looked for a nice way to reverse an int variable in python. I wanted to have a function which transforms for example "1234" into "4321". At the internet, i've found a nice math solution which worked flawless. It's code look's like:

    def rev(val):
      if val < 10: return val
      else:
        expon = int(math.log10(val))
        ival = val%10
        dval = val/10
        return int(ival*math.pow(10,expon)) + rev(dval)

    But today i tryed to make another, shorter solution. It has nothing todo with math. Actually i just take an int variable, convert it into string, reverse it, and convert it back into int. Really simple and it also works flawless:

    def strrev(sval):
      if len(str(sval)) == 1: return sval
      else:
        strcon = str(sval)
        return int(strcon[::-1])
    


    Now i'm wondering which function would be better and for what reason. I made some benchmarks and found out that the second solution is about 25% faster than the first one, but i just stoped these two version with "time". I think it's not good enough to take is as a benchmark. Besides that, i don't know anything about memory consumption. Anyway, i'll keep the second, since it's faster...

    If someone wants to try it out, i've uploaded both "benchmark" script's. The scirpt generates every number from 1 to 100000, reverses every number and subtracts every number with it's reversed version. 

    Download: Link

    Thursday, August 4, 2011

    alix update

    As i already mentioned in a earlier post, i'm having a alix device. It's a really cool device and i'm using it as my firewall and time-server, means it runs iptables and ntpd (there is also darkstat on it). The gentoo which is on it is a standard minimal installation, but with an hardened profile and hardened kernel. The firewall init-script is my own creation.

    Since this device is pretty slow and i also trying to keep the disk I/O low (the system is on a CF), i update it really rarely. Besides that, it has no vga output and i never could minicom (for the serial I/O) to work, i never know what's going on at the boot up and i can only hope everything works and sshd gets up :)

    Well, recently i made an update and it was an huge update. New gcc-compiler, new kernel and baselayout-2/openrc. I was really suprised that everything worked out flawless. I even have the feeling that my system boots up much faster with the new kernel and openrc. Really cool. Right after the update i took the CF and made a backup with dd.

    For those who still use such an alix device, i've uploaded the kernel config, so you can use it with your device..
    Download: gentoo-2.6.38-r6.config

    Monday, July 25, 2011

    tipp: eclean-dist with custom DISTDIR

    If you are using http-replicator you probably have all your distfiles in a seperate directory. Usually eclean-dist just cleans up the directory in the $DISTDIR variable, thus not in the directory where http-replicator stores the files. So how to clean this directory?

    Solution:
    DISTDIR="/home/michael/backups/distfiles/" eclean-dist

    Easy i know :) I put that into my crontab, so i never have to care about old packages.

    Wednesday, July 20, 2011

    custom dependencies in openrc

    Anyone ever searched for such a option? I did. The reason is simple. I have a really neat alix system with gentoo on it. The problem with this system is that it has no battery, means, every time you boot it up the time is set to 01012000. A good solution to update the time is via ntp-client and since its the gateway to the internet for my local network, it also has an ntpd damon for the local pcs.

    Now i also run darkstat on this alix device, just for fun to see the traffic. Well, i said every time i boot it up the time is set to 01012000. The darkstat init script usually always runs before the ntp-client and thus getting the date from 2000. It's a funny because if i look at the darkstat site it shows something like "running for 4234 days, since 2000-01-01". Though that would be awesome but unfortunately its wrong.

    A solution now would be to edit the darkstat init script and add to depend() "need ntp-client". It's a oneliner but it's not the correct way and with every update or reinstall of darkstat etc-update would overwrite it (though updates of darkstat are really seldom). Anyway there is a better (and correct) solution. Via /etc/rc.conf you can add/remove dependencies for any init scripts in gentoo.

    The correct onliner in /etc/rc.conf is rc_darkstat_after="ntp-client"

    That's all and it works perfectly.

    Tuesday, July 19, 2011

    new script: getlatestinstall

    I play alot around with kvm, thus i also need quite often the install-cds for gentoo. Unfortunately i never use the latest install-cds because i'm to lazy to download the media first. It's not a big problem, but i wanted to have a nice solution to have always the latest install-cds. Now i wrote a small script which does the downloads for me.

    The Features are:
    • check if the passed directories exist and if they are writeable
    • download the latest iso images from gentoo.distfiles.com + checksum files
    • check the md5sum of the iso-image, if failed delete the downloaded files, and try again (3 times)
    • if passed, create a link to the iso-image
    • works for amd64,x86,alpha and ia64
    I think it's quite useful and i put it into my cronjobs to run it every sunday. Maybe someone else can use it too.

    There is also a small discription included if you run the script without arguments. Don't forget to change the ${DOWNLOADDIR} and ${LINKSDIR} vars. Suggestions, bugs and improvements are welcome ;)

    Download: Link

    #!/bin/sh
    # Datei: getlatestinstall
    # Autor: Michael Mair-Keimberger (m DOT mairkeimberger AT gmail DOT com)
    
    # Copyright (C) 2011  Michael Mair-Keimberger
    #
    # This program is free software; you can redistribute it and/or
    # modify it under the terms of the GNU General Public License
    # as published by the Free Software Foundation; either version 2
    # of the License, or (at your option) any later version.
    # 
    # This program is distributed in the hope that it will be useful,
    # but WITHOUT ANY WARRANTY; without even the implied warranty of
    # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    # GNU General Public License for more details.
    # 
    # You should have received a copy of the GNU General Public License
    # along with this program; if not, write to the Free Software
    # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
    
    # Discription:
    # A script which automaticly download the latest
    # gentoo install cds for x86,amd64,alpha,ia64
    # checks the md5sum and make symbolic links to the iso
    
    # programs to use
    MD5SUM=/usr/bin/md5sum
    WGET=/usr/bin/wget
    LN=/bin/ln
    RM=/bin/rm
    CAT=/bin/cat
    TAIL=/bin/tail
    MKDIR=/bin/mkdir
    PING=/bin/ping
    
    # make sure both directory are writeable for the user
    # who execute this script
    LINKSDIR="/home/${USER}"
    DOWNLOADDIR="/home/${USER}"
    
    usage(){
      echo "$0 arch (downloaddir) (linksdir)"
      echo "arch  > select between amd64,x86,alpha or ia64"
      echo "downloaddir > directory where the iso will be stored (optional)"
      echo "linksdir > directory where the links will be stored (optional)"
      echo "A script to download the latest minimal cd for gentoo"
      exit 1
    }
    
    cfgchecks(){
      # check if the directories exists and if there are writeable
      for checkdir in ${DOWNLOADDIR} ${LINKSDIR}; do
        if [ -d "${checkdir}" ]; then
          if ! [ -w "${checkdir}" ]; then
     echo "Directory \"${checkdir}\" is not writeable!"
     return 1
          fi
        else
          echo "Directory \"${checkdir}\" dosen't exists"
          return 1
        fi
      done
    
      # create arch dirctory if it dosen't exists
      if ! [ -d "${DOWNLOADDIR}/gentoo-${1}/" ]; then
        ${MKDIR} ${DOWNLOADDIR}/gentoo-${1}/ > /dev/null 2>&1
      fi
    
      # check if there is internet connection
      if ! (${PING} -c1 -q distfiles.gentoo.org > /dev/null 2>&1); then 
        echo "No internet connection"
        return 1
      fi
    }
    
    md5sumcheck(){
      cd ${DOWNLOADDIR}/gentoo-${1}/
    
      if [ `${MD5SUM} -c --status ${DOWNLOADDIR}/gentoo-${1}/*.DIGESTS` ]; then
        ${RM} ${DOWNLOADDIR}/gentoo-${1}/*
        echo "md5sum check failed"
        return 1
      else
        # -f because if a link already exists it has to be forced
        ${LN} -s -f ${DOWNLOADDIR}/gentoo-${1}/*.iso ${LINKSDIR}/gentoo_${1}
      fi
    }
    
    download(){
      # download the txt file which points to the latest minimal cd
      ${WGET} -q -P/tmp/ http://distfiles.gentoo.org/releases/${1}/autobuilds/latest-install-${1}-minimal.txt
      LATEST_ISO=`${CAT} /tmp/latest-install-${1}-minimal.txt | ${TAIL} -n 1`
      ${RM} /tmp/latest-install-${1}-minimal.txt > /dev/null 2>&1
      ${RM} ${DOWNLOADDIR}/gentoo-${1}/* > /dev/null 2>&1
    
      # download the latest iso with the checksum files
      ${WGET} -q -P${DOWNLOADDIR}/gentoo-${1}/ http://distfiles.gentoo.org/releases/${1}/autobuilds/${LATEST_ISO} \
        http://distfiles.gentoo.org/releases/${1}/autobuilds/${LATEST_ISO}.CONTENTS \
        http://distfiles.gentoo.org/releases/${1}/autobuilds/${LATEST_ISO}.DIGESTS \
        http://distfiles.gentoo.org/releases/${1}/autobuilds/${LATEST_ISO}.DIGESTS.asc
    }
    
    main() {
      if cfgchecks $1; then
        for retry in {1..3}; do
          download $1
          if md5sumcheck $1; then
     break;
          fi
        done
      fi
    }
    
    
    if [ $# -lt 1 ]; then
      usage;
    else
      # change DOWNLOADDIR and LINKSDIR to user settings
      if [ "${2}" ]; then
        DOWNLOADDIR=${2}
      fi
    
      if [ "${3}" ]; then
        LINKSDIR=${3}
      fi
      main $1
    fi

    Monday, July 18, 2011

    webspace

    How to share files (pdf, doc, txt, anything) on blogger.com?

    Well, there is no way, since you can only upload pictures (on picassa). Recently i've looked for a easy way to share files, in particular i wanted to upload a script on my blog. Since it's only possible to upload pictures on blogger.com i searched for some webspace. The problem is, i don't wanted to create another account on any webhoster just for a few scripts. The scripts are really small and i would be already happy with 50MB.

    Luckily i found something and it's from google too. The solution is google sites! After creating a new "site" i'm able to upload content up to 100MB and the max files size can be 20MB. That's perfect for me! I think i never exceed those 100MB and even if i would, i can create new sites.

    After all, the solution is a bit stupid. It would be really kind of google if they would provide some webspace for blogger.com.

    Anyway - it works and now i'm able to share my scripts.

    How to create a new site: Link
    Storage features for a free site: Link
    How to upload content on google site: Link