I have written previously about BitTorrent Sync, the encrypted file syncing application that uses the bittorrent protocol to sync your data over your LAN, or over the Internet, using P2P technology. I have been using it since early this year as a replacement for dropbox.¹

At the beginning of this month, BitTorrent unveiled the Beta API for developers (meaning you have to tell them what you plan to do in order to be issued a key). After some equivocation, I signed up with the rather flimsy excuse of “writing a shell wrapper for the command line” and found, to my chagrin, a key in my inbox the next morning.

This proved to be something of an unwelcome arrival. In theory, I was excited about having access to a tool to query the Sync application. One of the nodes is on my headless Raspberry Pi and the idea of being able to issue a command from my laptop to ascertain what was going on in the Pi’s synced folders was (and is) a tremendously attractive one.

However, now that I was in possession of the key, I felt morally obliged to do something with it. The problem with this realization was that I had no idea how to work with an API, let alone writing a script to accomplish my purported goal.

After spending some time looking at the documentation, and buoyed by the optimism of ignorance, I decided to make good on my promise. My first attempt sort of worked, but was hampered as much by a serious conceptual flaw as it was by poor implementation. I decided, in spite of any number of Stack Overflow posts warning expressly not to do this, to use awk to parse the JSON data². This was what I would euphemistically describe as a “learning opportunity.” The result is preserved for posterity in my bitbucket repo (for the completists)…

At this point, I was fortunate that Earnestly in #archlinux introduced me to keenerd’s purpose built tool, Jshon. And by “introduced” I mean generously (and patiently) talked me through how it worked and how I could use it with the Sync API to achieve what I was after. After a while playing with it, there was the—inevitably belated—moment of realization: this thing is genius! It allows you to intelligently interrogate the data. Not blindly chopping at it with an increasingly complex series of actions in awk³; but quite directly traversing the structure and extracting the desired elements.

Now I was cooking. Well, more to the point, I was flailing about in a smoke filled kitchen convinced that the feeling of euphoria was inspiration—not imminent asphyxiation. Once again, Earnestly’s patience and bash skills were put to good effect. The resulting script, btstat, is undeniably a triumph of his good ideas over my own poor execution. In other words, the fact that it works so well is testament to his ability, but any and all faults are mine alone⁴.

And it does work. It only requires Jshon (which you will already have installed because you use aurphan, right?) and a file with your synced folders and their respective secrets on each line⁵, like so:

1
2

directory1 YourRe@llyTr1cky53cr3t
directory2 H0p3fu11yY0uG3tTh31d3@

The functionality in the script is limited at this stage to just querying the application; I wasn’t interested in pushing changes at this point. So you can access the version of the currently installed Syncapp, upload and download speeds, the size of synced folders and list all of their contents.

It is a beta release, so the odd bug is to be expected. Overall, though, the API is a welcome addition to what is a great application. If you have an API key, add it to your sync.conf, grab the script and give it a whirl. Undoubtedly, over the coming weeks more polished versions will emerge in “proper” languages, but for the time being this does exactly what I need.

Notes

1. Yes, I would much prefer to use an open source tool to accomplish the same thing but I haven’t found anything comparable that is this good to date…
2. In what I imagine is a complete coincidence, the JSON logo is uncannily similar to the initial (and much better, I thought) Sync logo, as seen on my original post.
3. This is not intended as a slight on awk; which I am undoubtedly fond of, but rather the limits of my ability with that language.
4. I am also indebted to Scott (firecat53) for testing and providing helpful feedback on the early versions of the script.
5. It is worth noting that in the $json variable, I pass curl the -n option, which means it reads my credentials from $HOME/.netrc.

For those people that prefer to forego the full blown DE and—instead of all the “convenience” that this sort of setup offers—piece together their setup from a variety of different tools and knit it all together with some scripting, automounting external drives is, notwithstanding the inexplicable number of posts to the forums to the contrary, incredibly straightforward with udisks. My approach in this respect is to use udiskie; it is lightweight, unobtrusive, configurable and foolproof, as far as I can tell.

Where I have found a small gap, or more a minor irritation really, is with unmounting devices. udiskie-umount /media/MY_USB_DRIVE works just fine, but I often have a variety of media mounted, and not just standard USB drives. At work, for example, it is not uncommon for me to have mounted under /media any or all of the following:

• A USB drive containing all of my music
• An encrypted volume mounted with tcplay shared via Bittorrent Sync
• A CIFS share
• An NFS share or an SSHFS mount.

In a couple of these cases, I don’t want to—or can't—just umount them with udiskie; they require a different approach. To facilitate this, I have adopted a simple approach: I use a standard naming convention for all external media or their mountpoints. When I first buy a USB drive, I give it a meaningful name, beginning with an uppercase letter, as this won’t clash with any of my internal drives and it happily accommodates drives from other operating systems. So:

1

sudo dosfslabel /dev/sdc1 EightBall

sets up a new 8GB drive I found in the schwag bag at the conference I attended last week.

Now that all external media are predictably named, it is just a matter of writing a script that checks how many are mounted, presents a menu if there is more than one, and unmounts the respective device correctly:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59

#!/usr/bin/env bash
# unmount USB drives

target=( $(awk '/media\/[\^A-Z]/ {print $3}' <(mount)) )
shares=(Scout Sentinel)

checkbusy() {
  grep "PID" <(lsof +d "$target" &>/dev/null)
  if [[ $? -eq 0 ]]; then
    printf "%s\n" "${target##*/} busy…"
    exit 1
  fi
}

exstatus() {
  if [[ $? -eq 0 ]]; then
    printf "%s\n" "${target##*/} unmounted…"
  else
    printf "%s\n" "Failed to unmount."
  fi
}

# check for multiple devices
if (( "${#target[@]}" > 1 )); then
  PS3="Select your device to unmount: "
  printf "%s\n" "There are ${#target[@]} devices mounted"
  select dev in "${target[@]}"; do
    target="$dev"
    break
  done
fi

# check for share
for drive in "${shares[@]}"; do
  if [[ "$drive" =~ "${target##*/}" ]]; then
    share="$drive"
  fi
done

# options per filesystem
if [[ -n "$target" ]]; then
  for drive in "${shares[@]}"; do
    if [[ "$drive" = "${target##*/}" && "${target##*/}" = Safebox ]]; then
      cmd=$(sudo safebox close)
    elif [[ "$drive" = "${target##*/}" ]]; then
      cmd=$(sudo umount "$target")
    else
      cmd=$(udiskie-umount -d "$target" &>/dev/null)
    fi
  done
# do it
checkbusy
$cmd
exstatus
else
  printf "%s\n" "No drive mounted!"
fi

# vim:set ts=2 sts=2 sw=2 et:

Now, no matter the number or type of devices I currently have mounted, I can unmount them by typing dismount and choosing the appropriate drive via the select menu. Simple, but satisfying. The script is in my bitbucket repo

Notes

Creative Commons image, Dismount, by Chris.

This is an addendum to my two previous posts on compiling kernels and automating the process. In the first of those posts I wrote about a wonderful tool to track the modules necessary for building a custom kernel with make localmodconfig, graysky’s modprobed_db. A simple bash script, modprobed_db allows you to build up an array of modules and—as the name suggests—modprobe all of them prior to compilation.

I forked it on github and started to play around with the script (mostly because I am slightly OCD about constructions like this:

1
2
3

cat /proc/modules | awk '{print $1}' | sort -k 1,1 | \
  grep -Ev "$(echo ${IGNORE[*]} | sed -e 's/^/^(/' -e 's/ /|/g' -e 's/$/)$/')" \
  >/tmp/.inmem

After hacking a while, I started to realize that the venerable UNIX programme Awk would, in many ways, be a better tool for this job. Essentially, we want to manage three lists: the modules in /proc/modules, a list of modules to ignore, and the master list of all modules required prior to compilation. This seems perfectly suited to Awk.

So, in the interests of learning Awk, I decided to rewrite the required functionality (or, more correctly, the subset that I required) in this language. This wasn’t a straightforward task for me. I got stuck a couple of times and, as I work in a building full of neckbeards, I thought I could ask for a couple of pointers. To my surprise, no-one I asked seemed interested in Awk. Typical responses were along the lines of “just use Perl.”

Aside from not knowing any Perl, and being unlikely to learn any in the immediate future, I was bemused by the notion that none of these skilled developers rated Awk as a language worth knowing. One of the #awk FAQ’s specifically addresses this: Why would anyone still use awk instead of perl? The quote by Tom Christiansen is worth repeating here:

Awk is a venerable, powerful, elegant, and simple tool that everyone should know. Perl is a superset and child of awk, but has much more power that comes at expense of sacrificing some of that simplicity.

In a happy coincidence, this question on Unix & Linux SE showed up just as I was completing my script and it neatly illustrates (to my unututored eye, anyway) an example of a typical situation where Awk’s strengths make it a better approach than Perl.

In any event, my rewriting of modprobed_db in Awk¹ resulted in this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

#!/usr/bin/awk -f
# script to manage modules for kernel builds

BEGIN { dbfile = ARGV[2] }

function red(s) {
    printf "\033[1;31m" s "\033[0m "
}

# read in the array
FILENAME != ARGV[3] {
  modlist[$1]++; next
}

# check for ignored modules
!modlist[$1]++ {
  print $1 >> dbfile
  close(dbfile)
  }

{
# modprobe modules
if (p == 1) {
  modload="sudo modprobe -a $(<"dbfile")"
  system(modload)
  close(modload)
  }
}

{
# update module count
if ( getline < dbfile ) {
  count++
  }
}

END {
  print red(count) "modules listed."
}

# vim:set ts=2 sts=2 sw=2 et:

Just prior to compiling a kernel, I can invoke this script with (the admittedly rather ungainly line):

1

awkmodules p=1 .config/kmod_db/ignored .config/kmod_db/modules_db /proc/modules

It does work, but I don’t claim that it is either idiomatic or attractive. The use of getline to update the module count strikes me as especially kludgy but I haven’t been able to think of a more correct way to handle it.

Incidentally, the gawk manual², Gawk: Effective AWK Programming has only just been updated³ so Awk is, unlike it’s homonym, clearly alive and well.

Notes

1. In my bitbucket repo.
2. The number of variants of awk is as much a delight as it is perplexing…
3. May, 2013

Public Domain image of the Great Auk by John James Audubon, from his book The Birds of America.