Thursday, September 1, 2011

FreeBSD 8.2, open-vm-tools and VMware Workstation

(A note for russian-speaking dudes: поцаны, прежде чем писать мне письма мелким почерком "у меня ни работаит шарные папки" или храбро скачивать х.з. какую версию порта откуда, откройте для себя 'vmware' тэг в левой колонке. Я верю что вы можете. На русский этот потс мне переводить лень, звиняйте панове.)

A custom port of open-vm-tools-2011.08.21-471295 for FreeBSD 8.2 can be obtained by typing:

% git clone https://github.com/gromnitsky/open-vm-tools-minimum.git

The port contains only a minimum version of VMware guest tools. It doesn't include kernel modules, hgfs client and wrappers for vmtoolsd.

Why?

  • The 'shared folders' option needs a guest support for vmblock file system and a hgfs kernel module. If one tries to say

    # mount -t vmhgfs .host:/shared-folder /mnt/hgfs
    

    He may get a kernel panic in FreeBSD 8.2. So, no shared folders for Jakucha.

  • vmblock is also required by drag&drop function provided by a vmusr/libdndcp.so plugin for vmtoolsd. Despite my attempts to repeat the struggle in 7.2, I couldn't achieve the same results with this version of open-vm-tools. So, no drag&drop and vmblock (fuse or .ko version) is useless.

  • I don't see a point of having external vmxnet driver instead of working em(4). And vmmemctl is like a furniture in the loft.

What have we in the so called minimum version?

  • Time sync.
  • Copy/paste under X.
  • VIX support.

The difference from previous versions:

  • vmware-user was removed and completely replaced by plugins for vmtoolsd utility.
  • GTK program vmware-toolbox was removed in favor of console version vmware-toolbox-cmd.
  • Unity plugin was excluded. It is too boring to explain why.

Installation

Delete all previous installations of open-vm-tools-minumum or open-vm-tools. This is mandatory, not optional.

After gitclonning cd to the open-vm-tools-minumum directory and decide: do you need X support for your virtual machine or it is just a console server? If the second variant is true, type:

# make WITHOUT_X11=1 install clean

otherwise

# make install clean

Then add to /etc/rc.conf line:

vmware_vmtoolsd_vmsvc_enable=YES

and type:

# /usr/local/etc/rc.d/vmware_vmtoolsd_vmsvc start

OK, now you have time sync and VIX.

If you've installed full variant with X support add this line to your ~/.xinitrc:

vmtoolsd -n vmusr &

(Don't miss the ampersand.)

Now if you start X (if X is already running type previous command in a xterm) you'll get copy/paste function between host and FreeBSD guest.

VIX Tests

The good news is: VIX works with FreeBSD guests and doens't require those kernel modules that were cutted.

To test VIX, we will use vmrun program (it comes with modern versions of Workstation). In Windows hosts it usually sit in C:\Program Files\VMware\VMware VIX directory.

Get a list of all running virtual machines:

>vmrun list
Total running VMs: 2
D:\vm\FreeBSD Stable\FreeBSD Stable.vmx
D:\vm\FreeBSD test\FreeBSD test.vmx

Get a list of all running processed in a virtual machine (it requires a name & a password of a particular user in the guest):

>vmrun -T ws -gu bob -gp asecretpass listProcessesInGuest \
 "D:\vm\FreeBSD test\FreeBSD test.vmx"
Process list: 50
pid=77251, owner=bob, cmd=/usr/local/bin/vmtoolsd -n vmsvc
pid=60007, owner=bob, cmd=/bin/csh
[...]

If vmrun hands for a long time there is a problem with vmtoolsd and its plugins in the guest. Check if vmtoolsd is running:

% ps -ax | grep vmtoolsd
  823  ??  S      1:02.33 /usr/local/bin/vmtoolsd -n vmsvc

And note the '-n vmsvc' option. Look into /var/log/messages file.

Something more interesting: copy a file from the host to the guest. This doesn't require any 'shared folders' or hgfs kernel modules. It works even if a network in the guest is down:

>vmrun -T ws -gu bob -gp asecretpass CopyFileFromHostToGuest \
      "D:\vm\FreeBSD test\FreeBSD test.vmx" d:\1.jpg "/tmp/1.jpg"

Cool, eh?

Cooking FreeBSD & Xorg under VMware Workstation 7.1

(The text below is for users who are new to configuring FreeBSD under VMware Workstation.)

Modern versions of Workstation automatically turn on emulation of Intel PRO/1000 network card if you choose 'FreeBSD' as an operating system while creating a new virtual machine. FreeBSD recognizes that device as em0. If it doesn't, turn off the virtual machine, close Workstation and insert ethernet0.virtualDev = "e1000" line into your .vmx file.

A long time ago mouse and video drivers were distributed only with a proprietry version of vmvare-tools. Today (it's like 4 years now) they are included with xorg and exist in FreeBSD ports collection. Install x11-drivers/xf86-video-vmware and x11-drivers/xf86-input-vmmouse ports.

The mouse configuring may be tricky. By default VMware Workstation emulates your mouse a ps/2 one with 5 buttons, disregarding if it a usual USB mouse on the host or a touchpad. You can force Workstation to exclusively grab the HID device (USB mouse) but then it will dissapear from the host. (VMware guest tools for MS Windows ship a special ps/2 driver which supports 6 & 7 mouse buttons (binded to back & forward in browsers) but I don't know any for FreeBSD.)

If you need mouse in syscons & X simultaneously, do not use moused option '-3' to emulate middle button for 2-buttons touchpad because you'll have problems with double click in X.

Xorg vmmouse driver (12.6.9) supports only 2 options (after Protocol & Device): Buttons and ZAxisMapping. Setting Buttons option to '7' won't bring you working 6 & 7 mouse buttons, though. (xev just ignores them.)

To turn on sound, type:

# kldload snd_es137x

and add to /boot/loader.conf file:

snd_es137x_load="YES"

If your laptop has the screen resolution 1280x800, add to xorg.conf section Monitor the line:

Modeline "1280x800" 70.00 1280 1312 1344 1376 800 801 804 850

Tuesday, August 30, 2011

FreeBSD 9 newfs block size and fragment size

In FreeBSD 9.0 beta1 2 new defaults for newfs(8) were introduced:

#define DFL_FRAGSIZE    4096
#define DFL_BLKSIZE     32768

Previous values (FreeBSD ≤ 8.2) were 2 times smaller. This means that by default new UFS filesystems will have 2 times less inodes.

This can be an unpleasant surprise if you want to quick install FreeBSD into a virtual machine with a small virtual hdd. For example, a quite modest setup with 4GB virtual hdd may bring you a /usr partition ~ 2.7GB. How many inodes is that?

The rough formula is:

10242s / 4f

where s is a partition size in MB and f is a magic number called fragment size. The new default is 4096 bytes (controlled by -f newfs(8) option).

So, for our example, the /usr partition will contain max 172,800 inodes. [1] Isn't that enough? (And why should I care?)

For a partitions larger than, say 6GB, you'll rarely ever bother with the word inode. But for smaller ones and such as /usr which usually holds full ports tree directory, you can put yourself into a strange situation when after a fresh FreeBSD install your /usr has ~ plenty of a free space but almost no free inodes.

Returning to our 2.7GB partition example, just after the installation and csup'ing of the ports tree, the number of files is:

# find /usr | wc -l
163114 

Or 94.4% of max possible files.

To make things worse, a new installer in FreeBSD 9.0 beta1 doesn't allow you to specify the fragment size & block size (b) values at all. The last one is important too, because the condition b/8 ≤ f ≤ b must be satisfied.

[1]After that number you'll get a dismal message from the kernel: /usr: create/symlink failed, no inodes free.

Monday, August 8, 2011

Emacs, xkb & xxkb

The song that never ends.

Every kid has its own scheme. Through the years I've picked out mine.

The prerequisites:

  1. You have tuned up xkb and know what key is your X layout switcher. (In the past I've used left Ctrl to toggle en/ru, but recently switched to capslock.)
  2. You're using xxkb as your X Window indicator of the current layout. Not from Gnome, not from KDE, etc.

We want: use capslock for emacs toggle-input-method too. We don't want to modify xorg system files or hack part of emacs written in C.

The problem is that emacs (under xorg-7.5.1, for example) ignores ISO_Next_Group completely. You cannot bind anything to the key that emits that event.

But there is a clever trick I've googled. You translate ISO_Next_Group to some unused key in emacs, for example Super-A. (Super == left win key on my machine.):

 (setf (gethash #xfe08 x-keysym-table) (aref (kbd "s-a") 0)) 

#xfe08 is ISO_Next_Group code. Then bind Super-A to toggle emacs keyboard layout:

 (global-set-key (kbd "s-a") '(lambda () (toggle-input-method))) 

OK. Next we need somehow disable system X switcher for all emacs windows. For xxkb this is done by adding this to its configuration file:

 XXkb.app_list.wm_class_class.alt_group1: Emacs 

And that's all.

As a bonus, I've added another very handy indicator of the current input method in emacs: cursor color.

;; default cursor color for all X frames
(add-to-list 'default-frame-alist '(cursor-color . "black"))

;; toggle the input method and the cursor color in one place
(global-set-key (kbd "s-a") '(lambda ()
                             (interactive)
                             (toggle-input-method)
                             (if window-system
                                 (set-cursor-color
                                  (if current-input-method
                                      "red3"
                                    "black")))
                             ))

Friday, August 5, 2011

Why pkg_version or portmaster -L Are Slow

Long time no see.

If you ever wondered why (in FreeBSD) any program that prints installed packages vs. available updates from the Ports is so freaking slow, then I have an answer for you and a maybe even a solution.

tl;dr: install pkg_noisrev gem. [1]

Why it is So Freaking Slow?

Just grabbing the installed list is simple: you read /var/db/pkg directory and you are done. The interesting part is how to check this data with the Ports tree.

Every correctly installed package records its origin in /var/db/pkg/package-name-1.2.3/+CONTENTS file. An origin is a relative path to the /usr/ports directory, for example for zip-3.0 package, the origin is archivers/zip.

Having that we can theoretically read corresponding Makefile (/usr/ports/archivers/zip/Makefile in our example) and compute the version number of the particular port. The problem is that 'the version' is a string that can containg 3 components (3 different Makefile varibles): port version, revision and epoch. Somethimes there is no port version but exists so called vedor version. Sometimes Makefile doesn't containt any version information at all but include (via a bsd make .include directive) another Makefile that can include another and so on.

So, to extract that information you need either:

  • Properly parse Makefile and recursively expand all its variables, read all includes, etc, i.e. write your own mini make utility.
  • Run "make -V PKGVERSION" command in the port directory.

You can guess what path was chosen by authors of the system pkg_version program or famous portmaster manager.

Think: run make for every package name; if you have 2,000 packages installed, make will run exactly 2,000 times.

To make thing worse, this is not the end of The Problem. Next quiz after obtaining the version number is how to compare 2 versions: the installed one and one from the Ports tree. They are not just simple numbers as in lovery rubygems. For example, what is newer: 4.13a or 4.13.20110529_1? Is there is a difference between 0.3.9.p1.20080208_7 and 0.3.9-pre1.20080208_7?

The system pkg_version utility contains a tedious comparation aloritm (/usr/src/usr.sbin/pkg_install/lib/version.c), reproduction of which is very boring. [2] So boring, that portmaster just calls "pkg_version -t v1 v2" to compare a pair of version number strings. Yes, if you have 2,000 packages installed, portmaster will execute make program 2,000 times + 2,000 times pkg_version program.

The last bit of slowness of such applications as pkg_version or portmaster is an iterative model. They read the whole packages list and process items one after another with 0 attempts to do things in parallel.

Can we do all that faster?

Yes, we can. A simple pkg_noisrev utility does that 4-5 times faster.

It tries to do a primitive parsing of Makefiles and if that fails only then executes make. It ships with the comparator extracted from pkg_version as a shared library that can be loaded once via Ruby dllopen method. It creates several threads and does things in parallel.

So, if you were running "portmaster -L" in the past, run "pkg_noisrev --likeportmaster" now.

[1]It requires Ruby 1.9.2.
[2]And, I dare to say, unsafe, because pkg_version doesn't have any tests you can reuse, so you can easily come up with some nasty bugs in your implementation.

Thursday, May 26, 2011

The Best of edw519

The famous book from one HN user can be read in 1-2 days (you can safely skip first 2 chapters.)

I've counted the number of interesting (to me) posts in the book: 57 of 256, or ~22.27%. Not bad.

The most hilarious chapter was #9.

Friday, May 13, 2011

Ruby Require Time

Last week in ruby-core mail list were circulating some horror charts of Ruby 'require' time for many .rb files. The most relevant is this one.

I can confirm the rather unpleasant statistics. I took the benchmark snippet from the link above and rewrote it into Rakefile. All it does is a creation of 18,000 empty files, and after that forces Ruby to 'require' them.

To run this benchmark on your machine, grab the source, place it into a file named Rakefile in a temporally directory, and type rake benchmark.

This is what I got in FreeBSD 8.2 under VMware Workstation 7.1.4:

$ rake clean benchmark # in RVM
(in /opt/samba/tmp/b)
rm -rf rec-files
mkdir -p rec-files
ruby 1.8.7 (2011-02-18 patchlevel 334) [i386-freebsd8.2]
              user     system      total        real
500 requires  0.171875   0.171875   0.343750 (  0.372954)
1000 requires  0.460938   0.453125   0.914062 (  1.006835)
1500 requires  1.039062   0.539062   1.578125 (  1.703206)
2000 requires  1.382812   0.632812   2.015625 (  2.186040)
2500 requires  2.953125   0.804688   3.757812 (  4.125621)
3000 requires  5.265625   0.625000   5.890625 (  6.766446)
3500 requires  7.609375   1.062500   8.671875 ( 10.501926)
4000 requires 11.039062   0.671875  11.710938 ( 12.816059)

versus scary:

% rake clean benchmark
(in /opt/samba/tmp/b)
rm -rf rec-files
mkdir -p rec-files
ruby 1.9.2p0 (2010-08-18 revision 29036) [i386-freebsd8]
              user     system      total        real
500 requires  0.156250   0.195312   0.351562 (  0.391295)
1000 requires  0.953125   0.390625   1.343750 (  1.459771)
1500 requires  3.171875   0.773438   3.945312 (  4.448573)
2000 requires  7.132812   1.390625   8.523438 ( 10.009462)
2500 requires 13.554688   1.859375  15.414062 ( 17.039073)
3000 requires 23.382812   2.976562  26.359375 ( 31.123249)
3500 requires 36.859375   3.734375  40.593750 ( 44.460490)
4000 requires 54.843750   5.453125  60.296875 ( 66.162403)
https://lh6.googleusercontent.com/_W-OHaMHyRAE/Tc2UUY8j7nI/AAAAAAAAARQ/GY2PZJGGQz4/s800/ruby-require-stat.png

Saturday, April 30, 2011

From Sinatra to Rails

... and from Sequel to ActiveRecord.

I don't know why but Rails 3.1 beta reminds me a Heffalump. Create a new project, uncomment in Gemfile 'edge Rails' lines and type bundle install --path vendor. And then type:

$ bundle list|grep '\* '|wc -l
   39
$ bundle list
Gems included by the bundle:
  * actionmailer (3.1.0.beta)
  * actionpack (3.1.0.beta)
  * activemodel (3.1.0.beta)
  * activerecord (3.1.0.beta)
  * activeresource (3.1.0.beta)
  * activesupport (3.1.0.beta)
  * ansi (1.2.3)
  * arel (2.0.7.beta.20110429111451 6330a18)
  * bcrypt-ruby (2.1.4)
  * builder (3.0.0)
  * bundler (1.0.12)
  * coffee-script (2.2.0)
  * coffee-script-source (1.0.1)
  * erubis (2.7.0)
  * execjs (0.3.0)
  * hike (0.7.1)
  * i18n (0.6.0beta1)
  * json (1.5.1)
  * mail (2.3.0)
  * mime-types (1.16)
  * multi_json (1.0.0)
  * polyglot (0.3.1)
  * rack (1.2.1 37d2b2f)
  * rack-cache (1.0.1)
  * rack-mount (0.7.2)
  * rack-ssl (1.3.2)
  * rack-test (0.5.7)
  * rails (3.1.0.beta cc35d5c)
  * railties (3.1.0.beta)
  * rake (0.8.7)
  * sass (3.1.1)
  * sprockets (2.0.0 ebd683e)
  * sqlite3 (1.3.3)
  * thor (0.14.6)
  * tilt (1.3)
  * treetop (1.4.9)
  * turn (0.8.2)
  * tzinfo (0.3.27)
  * uglifier (0.5.1)

'We are not amused.'

https://lh4.googleusercontent.com/_W-OHaMHyRAE/TbxkEsh1cbI/AAAAAAAAAQM/0om-5GfLtsU/s800/rails-and-sinatra.jpg

Thursday, April 7, 2011

Broken Printing in Internet Explorer 9

Уже думал у меня принтер навернулся, но все оказалось гораздо хуже.

Единственное применение IE, какое я за долгие годы нашел--генерация postscript для дальнейшего разбития .ps файла так, чтобы выходила двухсторонняя печать на принтере, который умеет только с 1 стороны. Разбитье происходит скриптиком на FreeBSD, но это не важно.

Важно то, что IE генерировал самые красивые страницы, на 2-м месте сидит Opera, потом Firefox, потом все остальное.

И тут на днях я что-то печатал с infoq, а принтер, отпечатав половину--затих и остановился. Что я только блин не делал. Тряс картридж, как шаман? Тряс. Выключал/включал принтер, как бухгалтерша? Включал/выключал. Конвертировал .ps в .pdf и пробовал печатать из Adobe Reader, как идиот? Конвертировал и пробовал. Нихера.

И только случайно проскролив в Reader'е до страницы, на которой заминка, обнаружилось, как:

  1. Reader ругается на отсутствие шрифта. Название шрифта самое неудобоваримое: какой-то случайный набор из [:alnum:].
  2. На той "плохой" странице в некоторых словах о сутст у т буквы.

"Ага", подумал я. "Вот вам бабушка и IE 9."

Пустив заранее от-thinapp'ленный IE 8 и сгенерировав .ps файл оттуда, выяснилось, что:

  1. 8-й генерирует нормальный postscript со всеми embedded шрифтами, как и раньше.
  2. Ренерируемый файл от IE 8 раза в 3 меньше чем от IE 9.
  3. Распечатать "плохой" вариант от 9-го можно только прогнав его (файл) через ps2ps. И такой от-distiller'ный файл принтер хоть и печатает, но задыхается--.ps файл получается каких-то совсем слонопотамовых размеров.

Короче говоря, сломали печать в IE 9, вот об чем я хотел рассказать.

Saturday, March 19, 2011

Серенькие будни Sequel

Если не смотреть на sql, который генерируют ORM, можно чувствовать себя хорошо. Если посмотреть на sql, который генерирует ORM, можно почувствовать себя очень хорошо, из-за того что не нужно получаемый sql ни набирать руками ни не дай б-г читать глазами.

Положим, у нас есть 2 отношения [1]: articles и tags. Обе модели которых, связанны как many-to-many. Отношения самые примитивные:

create_table(:articles) do
  primary_key :id
  String :title, null: false
  DateTime :updated, null: false
end

create_table(:tags) do
  primary_key :id
  String :name, null: false
end

create_table(:articles_tags) do
  primary_key :id
  foreign_key :tag_id, :tags
  foreign_key :article_id, :articles
end

Задачка: поискать в tags.name строчечьку foo и вывести связанные articles в обратном порядке. Для Sequel это выглядит так:

Article.order(:updated.desc).
  filter(id: DB[:articles_tags].select(:article_id).
   join(:tags, id: :tag_id).
   filter(:tags__name.like('%foo%'))).each {|i|
  puts i.title
}

Любой согласится--это гораздо читабельнее sql! Ну, ok, согласится, когда ему покажут отот sql:

#<Sequel::SQLite::Dataset: "SELECT * FROM `articles` WHERE (`id` IN
(SELECT `article_id` FROM `articles_tags` INNER JOIN `tags` ON
(`tags`.`id` = `articles_tags`.`tag_id`) WHERE (`tags`.`name` LIKE
'%mike%'))) ORDER BY `updated` DESC">

jfc.

[1]Если иногда вставлять это реликтовое слово в разговор посреди ланча, то после n-го по счету выговаривания вами отношения, добродушный собеседник начнет есть в (n-1)/3 раз медленнее.

Tuesday, February 15, 2011

Non-Recursive Rake

Для рубироидов: текст про нерекурсивный Rake и почему использовать его без рекурсии проще, про аллюзию на make и т.д., можно прочесть вот тут.

(Content-type там application/xhtml+xml из-за инлайновых svg, поэтому с IE туда заходить бесполезно.)

Cheers.

Wednesday, January 12, 2011

An advice

From Stroustrup's recent interview:

Danny Kalev: Finally, what are your New Years' resolutions?

Bjarne Stroustrup:

  • To get C++0x formally approved as an ISO standard.
  • To produce a good first draft of The C++ Programming Language (4th Edition).
  • To spend more time with my grandchildren.
  • To have at least one interesting new technical insight.

Dude, I have an insight for you:

Throw away your 28 years old crap into a wheelie bin & start using the sane, modern language: Go.

Such a brave but long-awaited move will:

  • liberate you from producing an invaluable 20,000 pages book;
  • automagically give an ability to spend more time with grandchildren.

With hope

AG