Et tu, Cthulhu

In addition to the kernel’s per-process CPU and I/O priorities, it would be nice to have memory residency priorities. That way, we could hint the kernel into keeping proportionally more pages of latency-sensitive desktop processes in RAM - like the main menu, the taskbar, the file manager and maybe some applets. Disk cache could have its own priority, a la the swappiness setting in /proc/sys/vm/swappiness. It could be a practical* way to mitigate the “my main menu takes several seconds from click to paint” problem.

* Since I’m not a kernel dev, I’m just guessing here.

Ben has some good news for anyone interested in reducing the memory usage of Linux programs. Smaps is great. But does it tell the whole truth?

For PCs with (basically) limitless swap space, it’s possible that it doesn’t even come close. The reason: Under memory pressure, only memory pages that are actually being accessed will be kept in main memory. These pages are usually 4kB each. So how much of a hog your process will be is determined by its access pattern. Which brings us to our dear old friend, Locality. It’s easy enough to deduce that a process with many tiny, frequently accessed memory blocks sprinkled in between “low urgency” blocks over a lot of pages will be a disaster compared to a process with frequently-accessed data allocated close together.

So, it’s possible that an even better measure of a process’ memory badness would be a running count of “number of different pages accessed in the last 10 seconds”, possibly with a falloff function (think load average). Maybe this could be done with a Valgrind tool, a la Cachegrind but for VM pages instead of CPU cache lines? Maybe it could even tell you where the N most frequently accessed memory blocks were allocated, and for each block, provide a list of locations where it was accessed (sorted by time last accessed or by frequency).

That sounds like a fun project!

Now that we know that even Evolution runs under Valgrind, we need a bigger challenge. So, how about the entire GNOME session?

I’ve written a couple of tiny scripts that lower the threshold to doing this. They take care of properly launching your session in Valgrind, collecting (and filtering slightly) the resulting logs and cleaning up (important, since you can get lots of lingering valgrind processes if you don’t).

I made an OpenSuSE package (for the 10.2 beta, possibly 10.1) that integrates this functionality as a pair of standard GDM sessions that you can select on login. Just click “Session” in the login screen and select one of the GNOME Valgrind ones, then log in using that. When you log out, it’ll generate a log file in “$HOME/valgrind-session.$N”.

The generated reports from a typical session say something about our code quality. Especially the leak report is interesting - the log file, after everything but “definite leaks” (i.e. allocated blocks to which no pointers exist, neither to the beginning nor to an internal offset) is removed, is about 2MB for a login + immediate logout here. Even though there are many repetitions and fairly harmless leaks, there are some serious-looking ones in there:

[~] grep “definitely” valgrind-session.0 | wc -l
1104

Just install and restart GDM. At least one billion bytes of RAM recommended to run.

Not everything works in an instrumented session (su, sudo definitely don’t, and I’ve had problems with “recent items” and logging out using the slab - remedied by adding a logout button to your panel), but overall it’s not bad. You can browse the web, read mail, use Nautilus, customize your desktop, launch applications (which will themselves be instrumented) etc.

If you like proactive bug fixing (and have fairly powerful hardware), I encourage you to check it out and maybe even improve on the concept (there’s a lot that could be done).

The Building Workload Characterization Tools with Valgrind talk slides, published just a couple of days ago, make it look easy.

You know the story. Random crashes preventing you from reading your mail all morning. This time, though, there’s a twist (and a moral).

The twist is that instead of complaining on IRC - ok, I mean in addition to complaining on IRC - I actually ran the crashy bugger through valgrind, much like you would run a zombie head through a blender. Sifting through the resulting goop provided me with enough information to file patches for buffer overrun 1, buffer overrun 2 and bug of the theoretical variety. All three bugs have been around for a really long time (several years).

As for the moral:

1) Valgrind works extremely well these days, even on large and complex programs like Evolution. It is nothing short of a masterpiece. It did not interfere with operation apart from the expected slowdown, and pinpointed the bug I was looking for (and then some) in a matter of minutes. It is highly recommended that programs be valground regularly with a “typical use” regimen, even if they appear to work fine. At the very least, this should be on all maintainers’ pre-release checklists.

2) If you’re a programmer, and a particular program is misbehaving for you, take the time to actually look for the bug. Valgrind makes it easy, and you’ll find trivial bugs even in large and complex programs. So there’s no reason to be intimidated. Even if you can’t immediately say what’s causing the problem, valgrind logs make for valuable bugzilla attachments.

3) Valgrind’s performance isn’t too bad, but it’s still the best excuse today for getting a faster computer. Start using it so you can justify the expense.

4) With a little time investment, Evolution is totally salvageable. If you were thinking of giving up on it, don’t. Version 2.8 has a tri-pane mail view and global search, making it an awesome mailer.

I just imported Flow to GNOME CVS (module “flow”). It has a fairly detailed HACKING file for those interested.

The low-level I/O is done (for Unix), modulo a little polish. Mid-level stream fundamentals are mostly done, but I need to write stream elements encapsulating all the low-level features. Work on the high-level easy-to-use interfaces has not yet started.

Events have had me thinking about computer games lately, so I thought I’d share. In case you’re looking for a Freed computer game classic, these are two underrated ones I’ve enjoyed:

Star Control 2 - Released for DOS in 1992, this is one of the few games I’ve played that actually provoked belly laughs at more than one occasion. Toys for Bob released the partially ported source under the GPL in 2002, to the delight of its persistent fan base. The game now runs pretty much perfectly on Linux, Windows and MacOS X, and fans have contributed additional artwork gravy.

Open Transport Tycoon Deluxe - This one is a real gem with a rough edge: The engine has been re-implemented from scratch and extended with tons of features including improved network play, but you need the artwork from the original game in order to play it. An effort is underway to produce high-quality replacement graphics - in the meantime, I’m told the original artwork is easy enough to “find”. This one is tri-OS too.

Oh, and don’t forget Toshok’s Starcraft clone! Although I’m not sure that’s playable yet. The SCSharp site seems to be broken at the moment.

In other news, Joakim is taking the Mexican media to task for its political bias and general hilarity. Hopefully he’ll keep this up with regular instalments - it’s funny because it’s true.

Refuse the shackles. Down with DRM! Pass it on.

If you have no idea what this means, go look at defectivebydesign.org, particularly their “about” page.

The lime tree in our garden keeps bearing large amounts of fruit.

I think we should just blow up Pluto.

This would put the poor bugger out of its misery and end the debate over whether it’s a planet or not. We’d learn a lot from the engineering challenges, and it’d be an interesting social experiment. Who’d chain themselves to Pluto and try to save it? Environmentalists? Astronomers?

Then someone could upload the movie to Youtube, and people would make billions of remixes - a Matrix version, a ping-pong ball version, the Star Wars version which would make it look like Alderaan and superimpose a giant laser on the footage, etc. Enterprising individuals would mix in their favourite musical scores and add subtitles in languages they don’t master. It’d be, literally, a blast.

And the best part is, I’m sure we could pull this off. Just convince the US administration that it’s time to think a little bigger. Make a statement. Show everyone who’s in charge. It’d make a nice little going-away present for the GWB too. You know, real mission accomplished stuff.