Tuesday, June 30, 2015

removing old backups

Ever had to think about how to remove old backups? Like you want to keep the last 30 backups? Whenever i had to think about a solution to this i though about something with "find -mtime". However, this only works when backups were made constantly on a daily base.

But what happens if a backup fails or an external server doesn't have a connection to the storage? In my case my laptop only sporadically creates backups. If my laptop would be turned off for 30 days all of my previous backups would be deleted with "find -mtime".

Until now i had a huge script which checks for such cases. Just stupid...

Today i found THE solution!
A really easy and nice one-liner to always keep the last 30 backups. It's just so nice :D

Attention: Don't simple copy/paste this one-liner - it can remove files without asking!

find /backup/folder -type f -printf '%T@ %p\n' | sort -k1 -n | head -n-30 | cut -d' ' -f2 | xargs -r rm

I think i don't really have to explain it. It keeps the last 30 backups - doesn't matter how old they are, but they are always the newest one. In case you have multiple backups make sure to keep them in separated directories or filter them with "find -name "*keyword*"". And before using this one-liner i strongly suggest removing the last part (xargs -r rm) to see what would be removed.

Hope someone can find it useful. I've searched hours to find something like this and never found anything. (probably because i searched with the wrong keywords...)

Monday, June 1, 2015

testing is fun (binpkg-multi-instance)

Since version 2.2.19 (now 2.2.20) portage implemented a feature called binpkg-multi-instance. This is a feature which i was looking for quite some time. In the last days i had some time and i decided to test it.
The feature itself brings the ability to portage to keep multiple versions (with different use settings) of a single package version.
Until now, if you created a binary package, portage could only keep exactly one binary-version of any package. If you build the package again with different use-settings and created a binary package the version prior would had gone.

Now this is probably not something many people were looking for. I was one of those who were really exited about it. When the feature hit git i was already tempted to test it directly from git head.

So why is that so exciting for me? Well, because some time ago i set up a nice test system where this feature helps _alot_ keeping compile times at a minimum.

Background:

The idea was simple. How to test many different setups and keep compile times at a minimum?
I wanted a base system which i could clone anytime i want so that i could install and test various package combination's and use already compiled packages as much as possible. That being said, a virtual machine with snapshots did come into mind. However, i had a dedicated hardware which had nothing to-do and thus there was no need for virtualization. Another candidate was btrfs with it's snapshot features. The problem here: What if i want to test another filesystem? ;)

Logically i decided to go with lvm.

The boot partition is shared with every system. Every other partition is on lvm. Every root partition is unique, only /home, /usr/portage and /usr/src are on a separate lvm partitions as those can be shared as well.
First i've created a "base" gentoo system. Basically a stage3 system with some additional programs and a few important settings.
EMERGE_DEFAULT_OPTS is one of the most important in this case. In my case it looks like following:

1
EMERGE_DEFAULT_OPTS="--binpkg-changed-deps=y --binpkg-respect-use=y --buildpkg --buildpkg-exclude \"virtual/* sys-kernel/*-sources\" -k --jobs=3"

It tells portage to always use, if possible, binary packages and, except for kernel-sources and virtual packages, to always create binary packages. Since this setting is my base system it's in every clone of it. (as long as i don't change anything by hand)

And that's were binpkg-multi-instance comes into mind. Since every system access the same binary package store, but every system might have different use setting for a particular package, every package now only has to build once in any case!

Compiling is really funny right now, cause it looks quite often similar like here:

Conclusion:

Sure, the whole setup is of course a bit more complex and while this setup works really great there are a few things to mentioned. For example, the kernel(s) needs a few features in every system (like lvm snapshot, openrc and systemd - if i want to test both, which i do). Also since home is shared with every system, testing various window managers (like kde,gnome,xlqt,...) could mess up their configurations. Also having different arches (x86 and amd64) need adjustments to the base configuration. (but it's working too!)

Besides that i've also wrote a small script which does most of the work. It clones and installs (grub) any system at any moment even with a different file-systems if desired. (plus it can also encrypt a cloned system).
For example, basically all i have to-do is:
1
./sysed -q
This clones the actual running system with the actual file-system and size and creates an grub entry which is called "${lvm_name}_testing".
The script can also backup, restore, delete and edit my lvm systems.

I'm using this script quite often as it's really simple cloning a whole system in about ~2 minutes. So far i already have 14 amd64 and 2 x86 systems. Below a list of my systems (from lvs).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
  gentoo_amd64_acp               vg0  -wi-a----- 10.00g
  gentoo_amd64_base              vg0  -wi-ao---- 10.00g
  gentoo_amd64_base_selinux      vg0  -wi-a----- 10.00g
  gentoo_amd64_base_systemd      vg0  -wi-a----- 10.00g
  gentoo_amd64_cinnamon          vg0  -wi-a----- 10.00g
  gentoo_amd64_enlightenment     vg0  -wi-a----- 10.00g
  gentoo_amd64_gnome             vg0  -wi-a----- 10.00g
  gentoo_amd64_kde               vg0  -wi-a----- 10.00g
  gentoo_amd64_kde_testing       vg0  -wi-a----- 10.00g
  gentoo_amd64_lxqt              vg0  -wi-a----- 10.00g
  gentoo_amd64_mate              vg0  -wi-a----- 10.00g
  gentoo_amd64_qtile_systemd     vg0  -wi-a----- 10.00g
  gentoo_amd64_sec               vg0  -wi-a----- 10.00g
  gentoo_amd64_secure            vg0  -wi-a----- 10.00g
  gentoo_x86_base                vg0  -wi-a----- 10.00g
  gentoo_x86_kde                 vg0  -wi-a----- 10.00g

binpkg-multi-instance had an big effect here, especially when trying things like abi_x86_32 or selinux. From now on i won't have to compile any package a second time anymore as long as i already build it once!

Big thx to the gentoo portage team!