jasonwryan.com

Miscellaneous ephemera…

Using Mercurial Queues

Over the last week or so, I have finally gotten around to digging into the whole concept of using Mercurial Queues to manage patches against an upstream project; in my case dwm.1 Essentially, this mercurial extension helps you manage a stack of patches on top of a directory tree. This gives you quite a lot of fine-grained control over your patchset and assists immeasurably with automating the application of patches when the underlying codebase changes.

After having played around with queues for the last couple of days, I am quite impressed: both the concept and the execution are simple and powerful. The concept of Mercurial Queues is best described in the Mercurial book: the aptly titled, Mercurial: The Definitive Guide:2

MQ's marriage of distributed revision control with patches makes it much easier to isolate your work. Your patches live on top of normal revision history, and you can make them disappear or reappear at will. If you don't like a patch, you can drop it. If a patch isn't quite as you want it to be, simply fix it—as many times as you need to, until you have refined it into the form you desire.

I followed Filippo Negroni’s excellent tutorial on the dwm site to setup my own dwm patchset based on tip, and the process was surprisingly straightforward; the only caveat being it is a good idea to plan quite carefully what changes you want in each particular patch. Once I had completed the setup, it got me thinking about the fact that the final install is done without pacman, just using make install. And, as a proof-of-concept more than anything else, I wondered whether this functionality could be used with makepkg.3

Turns out, as you might expect with Arch, it was not much of a job to set it up at all.

Note: this doesn’t mean I think it is a good idea; I was just interested to see if it could be done and how it would work once it was set up. For something like dwm, this is definitely a case of introducing a huge amount of complexity for no apparent benefit (for those people running Gnome, this may quite appeal to you…).

I have a working PKGBUILD that you can use if you want to follow along at home. Once you have downloaded it into dwm-hg/, run makepkg to clone the dwm repository and build the binary. Then you need to make sure that the mercurial queue extension is enabled, so your .hgrc needs to contain:

1
2
[extensions]
hgext.mq =

Now, you need to setup the queue repository. We do this in $srcdir, which is dwm-hg/src/dwm:

1
hg qinit -c

The next step is to start applying the various patches that you want to comprise your patchset on dwm. In my case that is base customizations to config.def.h and three other patches: statuscolours, cycle, and push. There are other patches in my repository, one of which patches the appropriate settings for an Arch build in the relevant makefiles. This is taken care of by the PKGBUILD in this case.

From here, it is very much as Filippo describes it in his tutorial: make some changes, add them to the queue and rinse and repeat:

1
2
3
4
hg qnew base.customizations
 # “hack hack hack…”
hg qrefresh
hg qcommit -m 'Added changes to config.def.h to customize'

After committing each of your changes as a discrete patch, you can review the queue with hg qseries:

1
2
3
4
5
6
┌─[Centurion ~/Build/dwm-hg/src/dwm]
└─╼ hg qseries
config.customizations
statuscolours
cycle
push

And hg qapplied will tell you which of the patches is currently applied in the working repository; at this stage the list should look the same as that in hg qseries. The next step is to remove all of the patches from the queue so that we have a clean repository:

1
2
3
4
5
6
7
┌─[Centurion ~/Build/dwm-hg/src/dwm]
└─╼ hg qpop -a
popping push
popping cycle
popping statuscolours
popping config.customizations
patch queue now empty

Should you need to edit a patch, it is simply a matter of popping to that spot in the stack (you can do so by name or by index number, beginning at 0):

1
2
3
4
5
6
7
8
9
┌─[Centurion ~/Build/dwm-hg/src/dwm]
└─╼ hg qpop config.customizations
popping push
popping cycle
popping statuscolours
now at: config.customizations
 # “hack hack hack…”
hg qrefresh
hg qpush -a

Once you are satisfied with the state of the stack, remove all of the patches to return to the original clean working directory and cd back to dwm-hg/. You can now use makepkg -fi to rebuild the package and you should see the pushed patches as part of the output:

1
2
3
4
5
6
7
8
==> Starting build()...
==> Pushing queued patchset
applying config.customizations
applying statuscolours
applying cycle
applying push
now at: push
dwm build options:

…and when you restart dwm, your customizations will be applied. When new changes are pushed to the dwm repo you can just makepkg -fi, and, if any of the patches fail due to changes in the underlying code, pop the stack to the failed patch, rebase the code, hg qrefresh and continue until done.

As I said at the beginning of this post, this isn’t a particularly smart way to use Mercurial Queues, PKGBUILDs or a combination of the two. You are much better off just cloning dwm to your local repository, initializing a queue repository and, once you are done setting up your patchset, issuing make && sudo make install.

If you have more than one machine, you can easily setup a repository for your patchset on bitbucket and host it there: then just pull the patchset queue from your other boxes and hg qpush -a. If you require a slightly different patchset for each box, you can use MQ to manage this process as well. This is where MQ really excels; I’ll cover this in more detail in my next post.

Notes

  1. See related dwm posts.
  2. I can’t recommend this {book,wiki} highly enough; it is an excellent example of thorough, accessible documentation.
  3. In the case of something like dwm, this is really irrelevant, as there is a single tiny binary and a man page, so keeping track of these files on your system is not an issue at all.

Flickr Creative Commons image by RachelH

The Myth of Breakage

Breakage is one of the memes of the Arch Linux community; it is referenced frequently on the Arch boards and in the various IRC channels. It doesn’t take newcomers long to realize that, whether something is actually broken or not, there is always someone they can blame.1

My experience of Arch, over the last three and a bit years, is quite the opposite. I have yet to experience anything that would qualify as breakage. Sure, there have been bugs, mostly upstream, and the more frequent moments where I haven’t been as diligent as I should, but not once have I had a machine that was unbootable or otherwise unusable. Arch has, on both my home and work machines, been incredibly stable and reliable.

For a number of Arch Linux users, however, that was not their experience over the last week when /lib was moved to /usr/lib. The volume of people on the boards and on IRC reporting unbootable or unusable systems was remarkable. Remarkable because, despite the fact that the news item, the wiki page, and the email to arch-announce all explicitly warned people not to use the --force option,2 so many did.

Sadly, these red-boxed warnings were apparently not enough. 90% of the breakage was because people either didn’t see, or disregarded, that advice. The remaining 10% were either in a hurry and didn’t pay close attention to how to deal with any leftover files in /lib or rm -rf‘ed in the wrong directory. Accidents happen.

What I am interested in is the people who fell into the much larger category, and principally those that didn’t see any of the communications about this change in advance of updating and reasoned, based on a recent change that did require the use of –force, that this was the most expedient way to resolve pacman reporting conflicting files.

Some context. The glibc update was released to the [testing] repository in the first week of July: and there were a couple of long threads3 about issues and fixes before it was pushed to [core]. By the time the package was released for general consumption, it had been tested and the developers had obviously planned how to make what is a fundamental system change with a minimal amount of user intervention. I think that, given the magnitude of the challenge, they did a superb job.

How then did it go so wrong for so many?4 To answer that, you have to consider the nature of Arch. It’s not the sort of distribution where you install it and then just go on your merry way, installing the least obsolete packages, revelling in the speed of the package manager and generally being a 1337 h@x0r…

Successfully running a rolling release like Arch, irrespective of your level of competence, means staying in touch with what is happening—in some form or other. Subscribing to the arch-general ML, visiting the forums, idling in IRC; somehow remaining connected to what is going on in the community so that you don’t blindly update one day and wonder why your system is broken.

This strikes me as a real strength; by design Arch encourages its users to participate in the community. Even if you are only lurking, over time you will inevitably find yourself posting in a thread, responding to an email, editing the wiki, adopting an orphaned package—increasingly getting involved and contributing.

This level of involvement is not for everyone. Some may find it onerous or a waste of their time. That’s fine. There are any number of other distros that do not ask for this level of commitment. The quid pro quo for running Arch, though, is that you are prepared, in whatever shape or form best works for you, to give something back; at the very least, that is your attention.

Notes

  1. This is, of course, a reference to said Arch meme: Allan is one of the developers who has contributed an enormous amount to Arch and the fact that his IRC nick is allanbrokeit is purely coincidental.
  2. At the end of 2011, the short option -f was removed from pacman so “it at least makes people read the word ‘force’ and maybe they will have some sense to step back and think about what they are doing first.”
  3. https://bbs.archlinux.org/viewtopic.php?id=144620 and https://bbs.archlinux.org/viewtopic.php?id=144616
  4. Relatively speaking. In situations like this, the overwhelming amount of traffic is about what went wrong, not people posting to say they had no difficulties whatsoever…

Creative Commons image on Flickr from djeucalyptus

Replacing Grub With Syslinux

With the announcement late last month that Grub 2.00 has been released I thought that it was time to stop procrastinating and do something about moving away from Grub Legacy, which is no longer actively developed. Not that there is anything wrong with Grub Legacy’s functionality; it has always done whatever I asked of it (admittedly, not a lot) and done so without complaint.

I have used Grub2 before, when I was running Ubuntu on my home desktop, and it is the bootloader on my Debian Squeeze server. I can’t, however, say that I have ever really understood—or appreciated—its mutliple file, “need to reconfigure after making any changes” type of complexity. So, I thought that I would give Syslinux a shot.

Syslinux isn’t a single bootloader, rather it is a collection of lightweight bootloaders, with Extlinux responsible for booting ext2, ext3, ext4 and btrfs filesystems. As the Arch Wiki Syslinux page notes, since Syslinux 4, Extlinux has been merged with Syslinux.

Installing Syslinux is ridiculously easy. It is literally a five step process. First, install the package:

1
pacman -S syslinux

After installing syslinux pacman will helpfully tell you what to do next:

==> If you want to use syslinux as your bootloader
==> edit /boot/syslinux/syslinux.cfg and run
==> # /usr/sbin/syslinux-install_update -i -a -m
==> to install it.

pacman says

Arch user Matthew Gyurgyik has written an ingenious script, syslinux-install_update that automates the process of writing the MBR and installing the relevant files to /boot/syslinux/. The script also checks if, as is the case with my desktop, there is a RAID array present, and if so passes the install command the --raid flag:

RAID mode. If boot fails, tell the BIOS to boot the next device in the boot sequence (usually the next hard
disk) instead of stopping with an error message. This is useful for RAID-1 booting.

man syslinux

After running the script, you should receive a message advising you that Syslinux has been sucessfully installed on both drives:

Detected RAID on /boot - installing Syslinux with --raid
Syslinux install successful
Boot Flag Set - /dev/sda1
Boot Flag Set - /dev/sdb1
Installed MBR (/usr/lib/syslinux/mbr.bin) to /dev/sda
Installed MBR (/usr/lib/syslinux/mbr.bin) to /dev/sdb

If you are moving from Grub Legacy, configuring Syslinux is similarly painless. There are two sections to a basic /boot/syslinux/syslinux.cfg, the defaults and the prompt entries.

I set up my defaults to be essentially invisible:

1
2
3
4
# General settings
DEFAULT Arch
PROMPT 0
TIMEOUT 0

And then copy across the relevant lines from my /boot/grub/menu.lst:

1
grep root= /boot/grub/menu.lst >> /boot/syslinux/syslinux.cfg

Then it is just a matter of adding the relevant entries under LABEL:

1
2
3
4
5
LABEL Arch
   MENU LABEL Arch Linux
   LINUX ../vmlinuz-linux
   APPEND root=/dev/mapper/vgroup-lvroot cryptdevice=/dev/md1:vgroup ro
   INITRD ../initramfs-linux.img

You then add entries for Fallback, and any other images that you may wish to boot from.

And that, in their entirety, is steps two and three. You then move to steps four and five:

1
2
pacman -Rsn grub
shutdown -r now

And you reboot with Syslinux. Done.

The beauty of Syslinux, from my perspective, is that it is simple to install and configure, with just the one config file and the options all well documented and easy to understand. If you want a menu with a background image, you can have that. If, like me, you just want to get on and boot your default, it does that.

Additionally, if you want or need to boot into Fallback, or want to dual boot, you can just hold down Shift at boot time and enter the desired Label at the prompt. Syslinux has Tab completion, so you only need enter a couple of letters, hit Tab and you will see all of the available Labels to boot. Brilliant.

Notes

Creative Commons Flickr image by custer_fluck