Tuesday, June 30, 2015

removing old backups

Ever had to think about how to remove old backups? Like you want to keep the last 30 backups? Whenever i had to think about a solution to this i though about something with "find -mtime". However, this only works when backups were made constantly on a daily base.

But what happens if a backup fails or an external server doesn't have a connection to the storage? In my case my laptop only sporadically creates backups. If my laptop would be turned off for 30 days all of my previous backups would be deleted with "find -mtime".

Until now i had a huge script which checks for such cases. Just stupid...

Today i found THE solution!
A really easy and nice one-liner to always keep the last 30 backups. It's just so nice :D

Attention: Don't simple copy/paste this one-liner - it can remove files without asking!

find /backup/folder -type f -printf '%T@ %p\n' | sort -k1 -n | head -n-30 | cut -d' ' -f2 | xargs -r rm

I think i don't really have to explain it. It keeps the last 30 backups - doesn't matter how old they are, but they are always the newest one. In case you have multiple backups make sure to keep them in separated directories or filter them with "find -name "*keyword*"". And before using this one-liner i strongly suggest removing the last part (xargs -r rm) to see what would be removed.

Hope someone can find it useful. I've searched hours to find something like this and never found anything. (probably because i searched with the wrong keywords...)

Monday, June 1, 2015

testing is fun (binpkg-multi-instance)

Since version 2.2.19 (now 2.2.20) portage implemented a feature called binpkg-multi-instance. This is a feature which i was looking for quite some time. In the last days i had some time and i decided to test it.
The feature itself brings the ability to portage to keep multiple versions (with different use settings) of a single package version.
Until now, if you created a binary package, portage could only keep exactly one binary-version of any package. If you build the package again with different use-settings and created a binary package the version prior would had gone.

Now this is probably not something many people were looking for. I was one of those who were really exited about it. When the feature hit git i was already tempted to test it directly from git head.

So why is that so exciting for me? Well, because some time ago i set up a nice test system where this feature helps _alot_ keeping compile times at a minimum.

Background:

The idea was simple. How to test many different setups and keep compile times at a minimum?
I wanted a base system which i could clone anytime i want so that i could install and test various package combination's and use already compiled packages as much as possible. That being said, a virtual machine with snapshots did come into mind. However, i had a dedicated hardware which had nothing to-do and thus there was no need for virtualization. Another candidate was btrfs with it's snapshot features. The problem here: What if i want to test another filesystem? ;)

Logically i decided to go with lvm.

The boot partition is shared with every system. Every other partition is on lvm. Every root partition is unique, only /home, /usr/portage and /usr/src are on a separate lvm partitions as those can be shared as well.
First i've created a "base" gentoo system. Basically a stage3 system with some additional programs and a few important settings.
EMERGE_DEFAULT_OPTS is one of the most important in this case. In my case it looks like following:

1
EMERGE_DEFAULT_OPTS="--binpkg-changed-deps=y --binpkg-respect-use=y --buildpkg --buildpkg-exclude \"virtual/* sys-kernel/*-sources\" -k --jobs=3"

It tells portage to always use, if possible, binary packages and, except for kernel-sources and virtual packages, to always create binary packages. Since this setting is my base system it's in every clone of it. (as long as i don't change anything by hand)

And that's were binpkg-multi-instance comes into mind. Since every system access the same binary package store, but every system might have different use setting for a particular package, every package now only has to build once in any case!

Compiling is really funny right now, cause it looks quite often similar like here:

Conclusion:

Sure, the whole setup is of course a bit more complex and while this setup works really great there are a few things to mentioned. For example, the kernel(s) needs a few features in every system (like lvm snapshot, openrc and systemd - if i want to test both, which i do). Also since home is shared with every system, testing various window managers (like kde,gnome,xlqt,...) could mess up their configurations. Also having different arches (x86 and amd64) need adjustments to the base configuration. (but it's working too!)

Besides that i've also wrote a small script which does most of the work. It clones and installs (grub) any system at any moment even with a different file-systems if desired. (plus it can also encrypt a cloned system).
For example, basically all i have to-do is:
1
./sysed -q
This clones the actual running system with the actual file-system and size and creates an grub entry which is called "${lvm_name}_testing".
The script can also backup, restore, delete and edit my lvm systems.

I'm using this script quite often as it's really simple cloning a whole system in about ~2 minutes. So far i already have 14 amd64 and 2 x86 systems. Below a list of my systems (from lvs).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
  gentoo_amd64_acp               vg0  -wi-a----- 10.00g
  gentoo_amd64_base              vg0  -wi-ao---- 10.00g
  gentoo_amd64_base_selinux      vg0  -wi-a----- 10.00g
  gentoo_amd64_base_systemd      vg0  -wi-a----- 10.00g
  gentoo_amd64_cinnamon          vg0  -wi-a----- 10.00g
  gentoo_amd64_enlightenment     vg0  -wi-a----- 10.00g
  gentoo_amd64_gnome             vg0  -wi-a----- 10.00g
  gentoo_amd64_kde               vg0  -wi-a----- 10.00g
  gentoo_amd64_kde_testing       vg0  -wi-a----- 10.00g
  gentoo_amd64_lxqt              vg0  -wi-a----- 10.00g
  gentoo_amd64_mate              vg0  -wi-a----- 10.00g
  gentoo_amd64_qtile_systemd     vg0  -wi-a----- 10.00g
  gentoo_amd64_sec               vg0  -wi-a----- 10.00g
  gentoo_amd64_secure            vg0  -wi-a----- 10.00g
  gentoo_x86_base                vg0  -wi-a----- 10.00g
  gentoo_x86_kde                 vg0  -wi-a----- 10.00g

binpkg-multi-instance had an big effect here, especially when trying things like abi_x86_32 or selinux. From now on i won't have to compile any package a second time anymore as long as i already build it once!

Big thx to the gentoo portage team!

Thursday, May 28, 2015

less portage rsync output

Ever thought how to silence rsync when doing emerge --sync (or eix-sync). Sure, it's nice when we get lots of information. But it's like with compiling packages - The first few years it's amazing looking at the terminal while it's compiling the latest stuff. However, after a while these things become a bit boring.
While we have --quite-build for emerge, rsync per default outputs every single file which gets transferred and deleted. Luckily, recent versions of rsync, which also went already stable, support new ways of progress output and since i use them already on other scripts i decided to modify my portage rsync settings a bit:

1
PORTAGE_RSYNC_EXTRA_OPTS="--info=progress2,name0,del0"

The output looks similar like that:


Neat, isn't it?
BTW, in order this works correctly - the remote rsync server need to run a recent version of rsync as well.

Monday, August 25, 2014

flashing android mobiles on gentoo

This is just a quick tip in case you ever want to flash a mobile phone on gentoo.

If you look at the cyanogenmod howto [1] (in my case for a nexus s) you'll see that you need the tools "adb" and "fastboot" which usually comes with the android sdk. Naturally the howto suggests you to install this sdk, which isn't even available on gentoo.
However if you don't want java and all it's other dependencies on your computer (which is required for the sdk) there is package which installs only those two needed tools. It's called dev-util/android-tools - and it's in portage :)

This is all you need:
* dev-util/android-tools
     Available versions:  (~)0_p20130123
     Homepage:            https://android.googlesource.com/platform/system/core.git/
     Description:         Android platform tools (adb and fastboot)

[1] http://wiki.cyanogenmod.org/w/Install_CM_for_crespo

Sunday, August 10, 2014

jumping directly into found results in menuconfig

For those who still use menuconfig for configuring their kernel - there's a neat trick which let you jump directly into a found result.

For example you would like to add a new driver. Usually you go into menuconfig and start searching for it with the "/" shortcut. What you probably not know, after you found your module - like you searched for the "NetXen Multi port Gigabit Ehernet NIC" with just searching for "xen" - you can go directly to the particular config via it's number shortcut:
Search result for "xen"












Notice this line:


The "(5)" is the shortcut. Just press the number 5 on your keyboard and you'll jump directly into the QLogic devices config.
For every found entry there is a number shortcut which let you directly jump into the given config. If you go back with esc-esc you also go back to the search result.

I think not many people know this trick and i hope someone can use it for further kernel builds ;)

Tuesday, August 5, 2014

kmscon - next generation virtual terminals

KMSCON is a simple terminal emulator based on linux kernel mode setting (KMS). It can replace the in-kernel VT implementation with a userspace console. It's a pretty new project and still very experimental.
Even though gentoo provides a ebuild its rather rudiment and it's better to use the live ebuild form [1] plus the libtsm package, which is needed for kmscon, from [2]. Personally i've added those ebuilds into my private overlay.

Don't forget to unmask/keyword the live ebuild:
# emerge -av =sys-apps/kmscon-9999

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild   R   *] sys-apps/kmscon-9999::local  USE="drm fbdev gles2 optimizations pango unicode -debug -doc -multiseat -pixman -static-libs -systemd" 0 kB

Total: 1 package (1 reinstall), Size of downloads: 0 kB

After successfully emerging kmscon it's pretty simple to start a new vt with (as root):
# kmscon --vt=8 --xkb-layout=de --hwaccel

This starts kmscon on vt8 with hardware-accel on and a german keyboard layout.

If your experimental you can add (or replace) an additional virtual terminal to your inittab. A line like following should suffice to start kmscon everytime you boot your system.
c11:2345:respawn:/usr/bin/kmscon --vt=8 --xkb-layout=de --hwaccel


I've tested it with my amd cards (r600g and radeonsi) and it worked with some minor output corruptions. However, in certain cases it works already faster than agetty, for example printing dmesg output. So far it looks really promising, sadly development seems to be really slow. You'll find the git repository here [3]

[1] https://bugs.gentoo.org/show_bug.cgi?id=490798
[2] https://bugs.gentoo.org/show_bug.cgi?id=487394
[3] http://cgit.freedesktop.org/~dvdhrm/kmscon/

Wednesday, April 2, 2014

howto - openvpn on gentoo

Today i gonna show you how to setup openvpn with self signed certificates and it's clients via cli or networkmanager (both using openvpn). I did made such setup a few days ago and i though i'll share my experience.

Server configuration:

Software:

I assume you have gentoo installed and running. Network should work too. Next we are going to install the needed packages. Depending on the openvpn version you also have to install easy-rsa. Openvpn prior to 2.3 have easy-rsa scripts included. I did install the latest unstable version, thus had to install easy-rsa as-well.
root # emerge -av openvpn
root # emerge -av easy-rsa

Since we need tun devices for openvpn you also have to make sure that tun devices are enabled in the kernel config (this is also needed on the clients):
root # cat /usr/src/linux/.config | grep CONFIG_TUN  
CONFIG_TUN=m


Certificates:

The scripts for generating the certificates are usually stored under /usr/share/easy-rsa/. Now edit following variables in the vars file: KEY_COUNTRY, KEY_PROVINCE, KEY_CITY, KEY_ORG, and KEY_EMAIL. Make sure these parameters are not left blank.

Edit vars file:
root # cd /usr/share/easy-rsa/
root # vim vars

Generate the ca file:
root # . ./vars  
root # ./clean-all  
root # ./build-ca  

The above sequence now defaults most parameters from the vars file. Only the common name has to be entered explicitly.

Generate the server certificate:
root # ./build-key-server server

Like in the previous step, most parameters are defaulted. When the Common Name is queried, enter "server". The last two queries require a positive responses:
Sign the certificate? [y/n]  
1 out of 1 certificate requests certified, commit? [y/n]

Generate client certificates:
root # ./build-key client1
root # ./build-key client2

Make sure using unique common names for each client. If you want password protected certificates use ./build-key-pass or if you want pkcs12 key files use ./build-key-pkcs12 instead. Again, the last two queries require a positive responses.

Generate Diffie Hellman parameters (needed by the server).
root # ./build-dh

Now we generated lots of files in the keys sub-directory. For the server we need following files: ca.crt, server.crt, server.pem and dh1024.pem

Now create a new folder in the openvpn configuration directory and copy those files into this folder:
root # mkdir -p /etc/openvpn/vpn  
root # cd keys  
root # cp ca.crt dh1024.pem server.crt server.key /etc/openvpn/vpn  


Configuration:

First open the server config:
root # vim /etc/openvpn/openvpn.conf

Below my example configuration:
An overview about all server configuration possibilities can be found at [1].
port 11194
proto tcp
dev tun
ca vpn/ca.crt
cert vpn/server.crt
key vpn/server.key
dh vpn/dh1024.pem
server 10.8.0.0 255.255.255.0
ifconfig-pool-persist ipp.txt
keepalive 20 120
comp-lzo
user nobody
group nobody
persist-key
persist-tun
verb 3
client-to-client
Some descriptions:
ca, cert, key, dh - options which should point to the certification files which we copied before. As seen on my example configuration you don't have to set the full path, just the relative path to /etc/openvpn/.
server - supplies a subnet range for the clients
client-to-client - vpn clients can "see" each other


Client configuration:

First you have to copy separately for every client following files from the keys directory (/usr/share/easy-rsa/keys) to the client (like via usb-stick): ca.crt, client1.crt and client1.key. Save it somewhere secure, ideally under /usr/openvnp/vpn.
root # mkdir -p /etc/openvpn/vpn
root # cp ca.crt client1.crt client1.key /etc/openvpn/vpn/


KDE-Networkmanager:

Make sure you have both networkmanagement and networkmanager-openvpn installed:
root # emerge -av networkmanagement networkmanager-openvpn

Next, Networkmanager:

Open Network Settings, switch to the VPN tab and add a new OpenVPN Connection.







Here you can give your connection an unique name. You also have to enter the Gateway which is the public ip address of your openvpn server. Also point to the right location of your ca, client certificate and client key file.

Under Opttional Settings you have to add the correct port of your server. Since openvpn runs on tcp with support for lzo compression you also has to check "Use LZO compression" and "Use TCP connection".










In the IPv4 Address tab you can add an additional DNS Server. This is useful if you have an local dns-server which is used to resolve local computer names.

If you don't want to have all trafic routed over the vpn tunnel, check "Use only for resources on this connection" under Routes.











That's all - now you can simple connect to your vpn with clicking on your vpn connection.


Openvpn init-Script:

Like any client you need to install openvpn first:
root # emerge -av openvpn

On Gentoo it's possible to create more tunnels by replacing VPN with other names. Each connection has its own configuration and can be stopped and started individually. The default is simply to use openvpn.conf and not symlink the service. You can of course use both methods. I'm going to show it with a separate openvpn configuration. First link the the new connection to the openvpn init-Script.
root # ln -s /etc/init.d/openvpn /etc/init.d/openvpn.VPN

Now create your config as /etc/openvpn/VPN.conf An overview about all client configuration possibilities can be found at: [2]
client  
dev tun  
proto tcp  
remote 1.2.3.4 11194
resolv-retry infinite
nobind  
user nobody  
group nobody  
persist-key  
persist-tun  
ca vpn/ca.crt  
cert vpn/client1.crt  
key vpn/client1.key  
comp-lzo  
remote-cert-tls server
Again ca, cert and key options are relative paths to /etc/openvpn.

After finishing the configuration you can start your openvpn connection with:
root # /etc/init.d/openvpn.VPN start

Done!


For the future it might be also interesting how to revoke someone's key. Below is a short howto for revoking certificates:

Revoking client certificates:

First switch do the easy-rsa directory:
root # cd /usr/share/easy-rsa/

Following command will generates a CRL file (crl.pem - certificate revocation list) and adds client's certificate to the revoke list.
root # . vars
root # ./revoke-full client

After doing so your output should be similar like:
Using configuration from /root/openvpn/20/openvpn/tmp/easy-rsa/openssl.cnf
DEBUG[load_index]: unique_subject = "yes"
Revoking Certificate 04.
Data Base Updated
Using configuration from /root/openvpn/20/openvpn/tmp/easy-rsa/openssl.cnf
DEBUG[load_index]: unique_subject = "yes"
client.crt: /C=KG/ST=NA/O=OpenVPN-TEST/CN=client/emailAddress=me@myhost.mydomain
error 23 at 0 depth lookup:certificate revoked

In order that openvpn is really going to drop connections from those certificates you have to add following to the server configuration.
crl-verify crl.pem

Make sure openvpn have access to this file. I suggest to copy this file directly to the openvpn configuration directory (/etc/openvpn)



Further help can be found here: Official Openvpn howto: https://openvpn.net/index.php/open-source/documentation/howto.html
Gentoo wiki openvpn guide: http://wiki.gentoo.org/wiki/OpenVPN
Revoking certificates: http://openvpn.net/index.php/open-source/documentation/howto.html#revoke
[1] https://openvpn.net/index.php/open-source/documentation/howto.html#server
[2] https://openvpn.net/index.php/open-source/documentation/howto.html#client