Tuesday, August 30, 2011

FreeBSD 9 newfs block size and fragment size

In FreeBSD 9.0 beta1 2 new defaults for newfs(8) were introduced:

#define DFL_FRAGSIZE    4096
#define DFL_BLKSIZE     32768

Previous values (FreeBSD ≤ 8.2) were 2 times smaller. This means that by default new UFS filesystems will have 2 times less inodes.

This can be an unpleasant surprise if you want to quick install FreeBSD into a virtual machine with a small virtual hdd. For example, a quite modest setup with 4GB virtual hdd may bring you a /usr partition ~ 2.7GB. How many inodes is that?

The rough formula is:

10242s / 4f

where s is a partition size in MB and f is a magic number called fragment size. The new default is 4096 bytes (controlled by -f newfs(8) option).

So, for our example, the /usr partition will contain max 172,800 inodes. [1] Isn't that enough? (And why should I care?)

For a partitions larger than, say 6GB, you'll rarely ever bother with the word inode. But for smaller ones and such as /usr which usually holds full ports tree directory, you can put yourself into a strange situation when after a fresh FreeBSD install your /usr has ~ plenty of a free space but almost no free inodes.

Returning to our 2.7GB partition example, just after the installation and csup'ing of the ports tree, the number of files is:

# find /usr | wc -l

Or 94.4% of max possible files.

To make things worse, a new installer in FreeBSD 9.0 beta1 doesn't allow you to specify the fragment size & block size (b) values at all. The last one is important too, because the condition b/8 ≤ f ≤ b must be satisfied.

[1]After that number you'll get a dismal message from the kernel: /usr: create/symlink failed, no inodes free.

Monday, August 8, 2011

Emacs, xkb & xxkb

The song that never ends.

Every kid has its own scheme. Through the years I've picked out mine.

The prerequisites:

  1. You have tuned up xkb and know what key is your X layout switcher. (In the past I've used left Ctrl to toggle en/ru, but recently switched to capslock.)
  2. You're using xxkb as your X Window indicator of the current layout. Not from Gnome, not from KDE, etc.

We want: use capslock for emacs toggle-input-method too. We don't want to modify xorg system files or hack part of emacs written in C.

The problem is that emacs (under xorg-7.5.1, for example) ignores ISO_Next_Group completely. You cannot bind anything to the key that emits that event.

But there is a clever trick I've googled. You translate ISO_Next_Group to some unused key in emacs, for example Super-A. (Super == left win key on my machine.):

 (setf (gethash #xfe08 x-keysym-table) (aref (kbd "s-a") 0)) 

#xfe08 is ISO_Next_Group code. Then bind Super-A to toggle emacs keyboard layout:

 (global-set-key (kbd "s-a") '(lambda () (toggle-input-method))) 

OK. Next we need somehow disable system X switcher for all emacs windows. For xxkb this is done by adding this to its configuration file:

 XXkb.app_list.wm_class_class.alt_group1: Emacs 

And that's all.

As a bonus, I've added another very handy indicator of the current input method in emacs: cursor color.

;; default cursor color for all X frames
(add-to-list 'default-frame-alist '(cursor-color . "black"))

;; toggle the input method and the cursor color in one place
(global-set-key (kbd "s-a") '(lambda ()
                             (if window-system
                                  (if current-input-method

Friday, August 5, 2011

Why pkg_version or portmaster -L Are Slow

Long time no see.

If you ever wondered why (in FreeBSD) any program that prints installed packages vs. available updates from the Ports is so freaking slow, then I have an answer for you and a maybe even a solution.

tl;dr: install pkg_noisrev gem. [1]

Why it is So Freaking Slow?

Just grabbing the installed list is simple: you read /var/db/pkg directory and you are done. The interesting part is how to check this data with the Ports tree.

Every correctly installed package records its origin in /var/db/pkg/package-name-1.2.3/+CONTENTS file. An origin is a relative path to the /usr/ports directory, for example for zip-3.0 package, the origin is archivers/zip.

Having that we can theoretically read corresponding Makefile (/usr/ports/archivers/zip/Makefile in our example) and compute the version number of the particular port. The problem is that 'the version' is a string that can containg 3 components (3 different Makefile varibles): port version, revision and epoch. Somethimes there is no port version but exists so called vedor version. Sometimes Makefile doesn't containt any version information at all but include (via a bsd make .include directive) another Makefile that can include another and so on.

So, to extract that information you need either:

  • Properly parse Makefile and recursively expand all its variables, read all includes, etc, i.e. write your own mini make utility.
  • Run "make -V PKGVERSION" command in the port directory.

You can guess what path was chosen by authors of the system pkg_version program or famous portmaster manager.

Think: run make for every package name; if you have 2,000 packages installed, make will run exactly 2,000 times.

To make thing worse, this is not the end of The Problem. Next quiz after obtaining the version number is how to compare 2 versions: the installed one and one from the Ports tree. They are not just simple numbers as in lovery rubygems. For example, what is newer: 4.13a or 4.13.20110529_1? Is there is a difference between 0.3.9.p1.20080208_7 and 0.3.9-pre1.20080208_7?

The system pkg_version utility contains a tedious comparation aloritm (/usr/src/usr.sbin/pkg_install/lib/version.c), reproduction of which is very boring. [2] So boring, that portmaster just calls "pkg_version -t v1 v2" to compare a pair of version number strings. Yes, if you have 2,000 packages installed, portmaster will execute make program 2,000 times + 2,000 times pkg_version program.

The last bit of slowness of such applications as pkg_version or portmaster is an iterative model. They read the whole packages list and process items one after another with 0 attempts to do things in parallel.

Can we do all that faster?

Yes, we can. A simple pkg_noisrev utility does that 4-5 times faster.

It tries to do a primitive parsing of Makefiles and if that fails only then executes make. It ships with the comparator extracted from pkg_version as a shared library that can be loaded once via Ruby dllopen method. It creates several threads and does things in parallel.

So, if you were running "portmaster -L" in the past, run "pkg_noisrev --likeportmaster" now.

[1]It requires Ruby 1.9.2.
[2]And, I dare to say, unsafe, because pkg_version doesn't have any tests you can reuse, so you can easily come up with some nasty bugs in your implementation.