jasonwryan.com

Miscellaneous ephemera…

Multi-arch Packages in AUR

One of the easiest ways to contribute to Arch is to maintain a package, or packages, in the AUR; the repository of user contributed PKGBUILDs that extends the range of packages available for Arch by some magnitude. Given that PKGBUILDs are just shell scripts, the barrier to entry is relatively low, and investing the small amount of effort required to clear that barrier will not only give you a much better understanding of how packaging works in Arch, but will scratch your own itch for a particular package and hopefully assuage someone else’s similar desire at the same time.

Now that I have a Raspberry Pi1, I am naturally much more interested in packages that can be built for the ARMv6 architecture; especially those that are available in the AUR. It is worth a brief digression to note that Arch Linux ARM is an entirely separate distribution and, while they share features with Arch, support for each is restricted to their respective communities. It is with this consideration in mind that I had begun to think about multi-arch support in PKGBUILDs, particularly in the packages that I maintain in the AUR.

I have previously posted about using Syncthing across my network, including on a Pi as one of the nodes. As the Syncthing developer pushes out a release at least weekly, I have been maintaining my own PKGBUILD and, after Syncthing was pulled into [community], I uploaded it to the AUR as syncthing-bin.

Syncthing is a cross platform application so it runs on a wide range of architectures, including ARM (both v6 and v7). Initially, when I wrote the PKGBUILD, I would updpkgsums on my x86_64 machine, build the package and then, on the Pi, have to regenerate the integrity checks. This was manageable enough for my own use across two architectures, but wasn’t really going to work for people using other architectures (especially if they are using AUR helpers).

Naturally enough, this started me thinking about how I could more effectively manage the process of updating the PKGBUILD for each new release, and have it work across the four architectures—without having to manually copy and paste or anything similarly tedious. Managing multiple architectures in the PKGBUILD itself is not particularly problematic, a case statement is sufficient:

PKGBUILD
case "$CARCH" in
    armv6h) _pkgarch="armv6"
            sha1sums+=('a94e5d00cec32956eb27bc12dbbc4964b68913f9')
           ;;
    armv7h) _pkgarch="armv7"
            sha1sums+=('9b782abf95668a906bfe76ad5ceb4cda17ec2289')
           ;;
    i686) _pkgarch="386"
          sha1sums+=('b2e1961594a931201799246f5cf61cb1e1700ff9')
           ;;
    x86_64) _pkgarch="amd64"
            sha1sums+=('035730c09ca5383c90fdd9898baf66b90acdef24')
           ;;
esac

The real challenge, for me, was to be able to script the replacement of each of the respective sha1sums, and then to update the PKGBUILD with the new arrays. Each release of Syncthing is accompanied by a text file containing all of the sha1sums, each on its own line in a conveniently ordered format, like so:

sha1sums.txt.asc
b2e1961594a931201799246f5cf61cb1e1700ff9    syncthing-linux-386-v0.9.16.tar.gz
035730c09ca5383c90fdd9898baf66b90acdef24    syncthing-linux-amd64-v0.9.16.tar.gz
d743b64204f0ac7884e4b42d9b1865b2436f5ecb    syncthing-linux-armv5-v0.9.16.tar.gz

This seemed a perfect job for Awk, or more particularly, gawk’s switch statement, and an admittedly rather convoluted printf incantation.

    switch ($2) {
      case /armv6/:
        arm6 = $1
        break
      case /armv7/:
        arm7 = $1
        break
      case /linux-386/:
        i386 = $1
        break
      case /linux-amd64/:
        x86 = $1
        break
      }
  }
END {
  printf "case \"$CARCH\" in\n\t"\
         "armv6h) _pkgarch=\"armv6\"\n\t\tsha1sums+=(\047%s\047)\n\t\t;;\n\t"\
         "armv7h) _pkgarch=\"armv7\"\n\t\tsha1sums+=(\047%s\047)\n\t\t;;\n\t"\
         "i686) _pkgarch=\"386\"\n\t\tsha1sums+=(\047%s\047)\n\t\t;;\n\t"\
         "x86_64) _pkgarch=\"amd64\"\n\t\tsha1sums+=(\047%s\047)\n\t\t;;\n"\
         "esac\n",
         arm6, arm7, i386, x86
}

The remaining step was to update the PKGBUILD with the new sha1sums. Fortunately, Dave Reisner had already written the code for this in his updpkgsums utility; I had only to adapt it slightly:

excerpt from updpkgsums
{
  rm "$buildfile"
  exec awk -v newsums="$newsums" '
    /^case/,/^esac$/ {
      if (!w) { print newsums; w++ }
        next
      }; 1
      END { if (!w) print newsums }
  ' > "$buildfile"
} < "$buildfile"

Combining these two tasks means that I have a script that, when run, will download the current Syncthing release’s sha1sum.txt.asc file, extract the relevant sums into the replacement case statement and then write it into the PKGBUILD. I can then run makepkg -ci && mkaurball, upload the new tarball to the AUR and the two other people that are using the PKGBUILD can download it and not have to generate new sums before installing their shiny, new version of Syncthing. You can see the full version of the script in my bitbucket repo.

Notes

  1. See my other posts about the Pi

Creative Commons image of the Mosque at Agra, by yours truly.

Simple Reminders

Due to a rather embarrassing episode in #archlinux a couple of weeks ago, where I naively shared one of the first bash scripts I had written without first looking back over it1, and had to subsequently endure what felt like the ritual code mocking, but was in fact some helpful pointers as to how I could make the script suck less (a lot less) I have been going through those older scripts and applying the little knowledge that I have picked up in the interim; reappraising the usefulness of the scripts as I go.

One that has proved to be of some utility for many years now is a simple wrapper script I wrote to help manage my finances. Like many useful scripts, it was written quickly and has been in constant use ever since; becoming almost transparent it is so ingrained in my workflow.

The script allows me to manage the lag between when a company emails me an invoice and when the payment is actually due. I find that companies will typically email their invoices to me some weeks in advance, whereupon I will make a mental note and then, unsurprisingly, promptly forget all about it, thereby opening myself up for penalties for late payment. It didn’t take me long (well, in my defence, a lot less time than it took for invoices to become digital) to realise that there was a better way™ - a script.

The at command is purpose built for running aperiodic commands at a later time (whereas cron is for periodic tasks). So, using at(1), once I receive an invoice, I can set a reminder closer to the final payment window, thereby avoiding both the late payment penalty—and the loss of interest were I to pay it on receipt. I just needed a script to make it painless to achieve.

The main function of the script is pretty self-explanatory:

todo
aread() {
  read -p "Time of message? [HH:MM] " attime
  read -p "Date of message? [DD.MM.YY] " atdate
  read -p "Message body? " message

  timexp='^[0-9]{2}:[0-9]{2}'
  datexp='^[0-9]{2}.[0-9]{2}.[0-9]{2}'

  if [[ $attime =~ $timexp && $atdate =~ $datexp ]]; then
     at "$attime" "$atdate" << EOF
     printf '%s\n' "$message" | mutt -s "REMINDER" jasonwryan@gmail.com
EOF
  else
     printf '%s\n' "Incorrectly formatted values, bailing..." && exit 1
  fi
}

Now, an invoice arrives, I open it and fire up a scratchpad, and follow the prompts. A couple of weeks later, the reminder email arrives and I login to my bank account and dispatch payment. You could, of course, have the script trigger some other form of notification, but an email works well for me.

The rest of the script is similarly basic; just some options for listing and reading any queued jobs and some more rudimentary checking. The full script is in my bitbucket repo2.

Update 7/09/14

Not more than a couple of hours after posting this, Florian Pritz pinged me in #archlinux with some great suggestions for improving the script. I particularly liked relying on date(1) handling the input format for the time and date values. He also suggested a readline wrapper called (appropriately enough) rlwrap and a tmpfile to better manage input validation. You can see his full diff of changes. In the end, I adopted the date suggestion but passed on rlwrap. Thanks for the great pointers, Florian.

Notes

  1. In the interests of full disclosure, the most egregious line was myterm=$(echo $TERM) which I would hope I copied blindly from somewhere else, but accept full responsibility for nonetheless.
  2. Don’t poke around too much in there, I still have quite a lot of cleaning up to do…

Creative Commons image by Adelle and Justin on Flickr.

Building from Source

One of the real strengths of Arch is its ability to be customised. Not just in terms of the packages that you choose to install, but how those packages themselves can be patched, altered or otherwise configured to suit your workflow and setup. I have posted previously about, for example, building Vim or hacking PKGBUILDS. What makes all this possible is the wonderful ABS, the Arch Build System.

Essentially a tree of all of the PKGBUILDs (and other necessary files) for the packages in the official repositories, the ABS is the means by which you can easily acquire, compile and install any of the packages on your system:

ABS is made up of a directory tree (the ABS tree) residing under /var/abs. This tree contains many subdirectories, each within a category and each named by their respective package. This tree represents (but does not contain) all official Arch software, retrievable through the SVN system.

Arch Wiki ABS

I have been using ABS since I started running Arch and it has worked well. I wrote a simple script to check for and download updates when required to help simplify the process and have been generally content with that approach. That isn’t to say that elements of this process couldn’t be improved. One of the small niggles is that the ABS only syncs once a day so there is almost always—for me down here in .nz, anyway—at least a full day’s wait between the package hitting the local mirror and the updated ABS version arriving. The other issue is that you download and sync the entire tree…

That all changed when, at the start of this month, one of the Arch developers, Dave Reisner, opened a thread on the Arch boards announcing asp, the Arch Source Package management tool, a git-based alternative for abs1.

Basically a 200-line bash script, asp is an improvement over abs insofar as you get the updated PKGBUILDs immediately; you can choose between just pulling the necessary source files (as per abs), or checking out the package branch so that you can create your own development branch and, for example, keep your patch set in git as well.

You can elect to locate the local git repository in a directory of your choosing by exporting ASPROOT, there are Tab completion scripts for bash and zsh and a succinct man page. Overall, for a utility that is only three weeks old, asp is already fulfilling the function of a drop-in replacement; a faster, more flexible tool for building Arch packages from source.

With thy sharp teeth this knot intrinsicate
Of life at once untie…

Antony and Cleopatra V.ii

Notes

  1. The package, not the entire build system…

Creative Commons image, Red Lego Brick by Brian Dill on Flickr.