Vim is not just an editor (and not in the way that Emacs is more than just an editor); it is for all intents and purposes a universal design pattern. The concept of using Vim’s modes and keybinds extends from the shell through to file managers and browsers. If you so choose (and I do), then a significant amount of your interactions with your operating system are mediated by Vim’s design principles. This is undeniably a good thing™ as it goes some way to standardising your command interface (whether at the command line or in a GUI).

To that end, I have been spending more time working with Vim and trying to improve both my understanding of it’s capabilities, the environment in which I use it and how I can optimise both of those conditions to not necessarily just be more productive, but to minimise friction in my work flows.

A large part of my job involves writing and, happily, so does a good deal of what constitutes my leisure activity. Whether it is emails written in Mutt, these blog posts, longer documents written in LaTeX, or just roboposting to forums; it is Vim all the way down. So I have spent some effort setting up Vim to make that experience as comfortable as possible.

The first step, and one I took several years ago now, was to write custom colour schemes, depending on whether I am in the console or X. Several weeks ago, I came across a Vim plugin called DistractFree, which is described as “enabling a distraction free writing mode.” I had always been slightly (well, perhaps scathingly would be more accurate) cynical when reading about these sorts of things when they first started to appear, but—after playing with two or three of them—this one has really grown on me (see the screenshot on the right).

I adapted my colour scheme for it, added a line to my ~/.vimrc to enable it for specific document types, and have not looked back.

1

autocmd BufRead *.markdown,*tex call DistractFree#DistractFreeToggle() | wincmd w

The final piece was to set up spell checking to work properly. As well as using English, I wanted to share my personal whitelist of words between all of my machines, so I define a location in ~/.vimrc as well:

1
2
3

set spelllang=en_gb                               " real English spelling
set dictionary+=/usr/share/dict/words             " use standard dictionary
set spellfile=$HOME/Sync/vim/spell/en.utf-8.add   " my whitelist

Then it is just a matter of ensuring that misspelled or other flagged words (like those that should be capitalised) are highlighted correctly. This required a couple of lines in my colour schemes:

1
2
3
4
5
6
7
8
9
10
11

" Spell checking  --- 
if version >= 700
  hi clear SpellBad
  hi clear SpellCap
  hi clear SpellRare
  hi clear SpellLocal
  hi SpellBad    ctermfg=9
  hi SpellCap    ctermfg=3    cterm=underline
  hi SpellRare   ctermfg=13   cterm=underline
  hi SpellLocal  cterm=None
endif

The most significant change, however, was that I recently purchased a hard copy of Drew Neill’s excellent Practical Vim. I have long been a fan of Drew’s excellent podcast, Vimcasts, and the book is every bit as impressive as the collection of those episodes, and more. Reading it on my commute over the last week, I have been constantly amazed at the power and flexibility of this editor and Drew’s ability to clearly articulate how to work economically and efficiently with it. I’m convinced this is an invaluable resource for anyone that currently uses Vim or wants to learn.

I expect that, over time, as I become more proficient with Vim, that I will adapt some of these settings, but for the time being they feel very comfortable and allow me to focus on hacking words together (and to do the occasional bit of scripting on the side)…

Notes

Image is from the cover of Drew Neill’s Practical Vim, by Ben Cormack.

Using a Socks proxy over an SSH tunnel is a well documented and simple if much less flexible stand in for a full-blown VPN. It can provide a degree of comfort when accessing private or sensitive information over a public Internet connection, or you might use it to get around the terminally Canutian¹ construct that is known as geo-blocking; that asinine practice of pretending that the Internet observes political boundaries…

By way of a digression, it occurred to me at some point while I was wrestling with setting this up that, over the last seven or so years, much of the “entertainment” provided by corporate content distributors has been in the form of encouraging me to spend hundreds? thousands? of hours researching and implementing ways to circumvent their litany of failed and defective technological restrictions: region codes, DRM and the like. It is worth noting that, in the vast majority of cases, I was just seeking access to content that I already owned (in another format), or was prepared to pay for.

My move to GNU/Linux in 2007 was in large part motivated by the awful realisation that the music I had bought in iTunes was stuck in there. The combined intellectual effort globally expended trying to legitimately route around broken copyright law would have comfortably powered another golden age of the sciences; it’s not entirely implausible to think that the only reason we still have to deal with cancer is the malignant legacy of Sonny Bono².

Now, back to our regular programming… One of my approaches to get around this sort of economic and policy rigor mortis has been to use a basic script to create a proxy tunnel to my home server. It assumes that you have public key authentication set up and your passphrase loaded in keychain, or something similar.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

#!/usr/bin/env bash

SSH_HOST="jason@XXX.XXX.XXX.XXX -p XXX -i $HOME/.ssh/box1"

up(){
    ssh -f -N -D 8080 -M -S /tmp/ssh_tunnel_%h.sock -o ExitOnForwardFailure=yes $SSH_HOST && \
    printf '%s\n' "ssh tunnel started successfully" || \
    printf '%s\n' "ssh tunnel failed to start"
}

down(){
    ssh -S /tmp/ssh_tunnel_%h.sock -O exit $SSH_HOST
}

if [[ $1 = up ]]; then
    up && chromium --proxy-server="socks://127.0.0.1:8080" &
elif [[ $1 = down ]]; then
    down
else
    printf '%s\n' "Fail…"
    exit 1
fi

Over the last couple of weeks, though, while I have been setting up and playing with Syncthing, I found this script wanting. With six nodes and, depending if I was on the LAN or not, as many as four of those hosts only accessible via SSH, then having the ability to quickly and painlessly open a browser on any one of the nodes without having to edit the script suddenly seemed like quite a good idea.

Accordingly I went to work on the script, including a test to determine if I was on my home network and passing the name of the desired host as an argument. With this approach, I simply type tunnel $host and chromium opens tunneled to that host, where I can the happily open Hulu the Syncthing GUI.

The updated script is posted as a gist, and as you can see, still needs some work to make it a little more generic. You will need, for example, to hand edit in the hosts and ports in get_host(). It is also the first time I have played with named pipes and I am not convinced that my use of mkfifo here is either the correct approach or implementation; but it works. Comments enlightening me would be gratefully received.

Notes

1. The good king was, appropriately enough, actually called Cnut the Great…
2. And, no, I am not referring to his musical corpus, which is as carcinogenic as his political career was as a Myrmidon for Big Content.

Flickr Creative Commons image, The Tunnel, by Lawrence Whitmore.

Just over a year ago, I was invited to participate in an exciting Alpha, for BitTorrent Sync. I wrote up my experience and, with one reservation, found it to be a fantastic tool that did pretty much everything I needed, including letting me finally get rid of Dropbox. That one reservation, although not entirely clear at the time, was clarified soon after and was crushingly disappointing: the programme was closed source.

Due to a combination of inertia (it was installed on all my devices and working well) and the lack of an alternative that didn’t involve installing and running a LAMP stack just to sync my files, I have persisted with it; even writing a simple Awk script to interact with the API. For the last twelve months I have been pretty happy with it.

In addition to the licensing choice, there are some other small issues that I have discovered: the Android App chews through your battery if it is left on (which sort of defeats the purpose, but I learned to live with it), the syncing can take a little while to pick up changes, and nodes that don’t see much action may flake out completely after a while.

Then, a few weeks ago I read about clearskies, an open source implementation of this concept. Sadly, it is still a proof of concept. This discovery, however, led to an extremely fortuitous exchange on Twitter with cayetano santos who pointed me at Syncthing. Described thus (cue background music):

Syncthing replaces Dropbox and BitTorrent Sync with something open, trustworthy and decentralized. Your data is your data alone and you deserve to choose where it is stored, if it is shared with some third party and how it's transmitted over the Internet.

Written in Go, Syncthing is still in early development, but judging by the release history, it is moving ahead quickly. The developer, Jakob Borg, is obviously both talented and productive. After using it for the last couple of weeks, I have uninstalled BitTorrent Sync from all my machines and have Syncthing set up and running nicely. It is simple, elegant and works wonderfully; for what it does.

Which is to say that it is not all of the things that either of its predecessors are: it isn’t intended to distribute files, or to host images or simple websites. It doesn’t include versioning or have a built-in editor. It is for securely syncing your files independent of any third parties. And there is no Android app, so your battery is safe¹.

Setup is through a tidy, bootstrap built web interface, by default running on 127.0.0.1:8080. I subsequently found that once I understood how the configuration worked, editing the config file at $XDG_CONFIG_HOME/syncthing/config.xml was much simpler. When setting up, you will also do a lot of stopping and starting of syncthing; if you use the service file that comes with the AUR package, that won’t work so well from the web interface… Run it from the command line with one of the STTRACE variables to debug and then, once it is running to your liking, hand it off to systemd².

With a reasonably tricky cluster setup (six machines; two outside the LAN sharing half a dozen repositories with three different topologies), there is a fair bit of trial and error involved getting everything set up and working correctly. But when you do, it is much faster to pick up changes than either of the other two and there is none of the risk around privacy and security presented by Dropbox and—to a (much, I would hazard to guess) lesser extent—BitTorrent Sync.

I have also found that it is much quicker for all of the nodes connect if, for the LAN connections in your cluster, you specify the IP addresses and port and disable global discovery for those nodes: this speeds up the discovery time when the nodes are started. Nodes still, from time-to-time, flake out, particularly if one of them is rebooted. Restarting the offending node is usually sufficient to bring the cluster back into sync.

That isn’t to say that there aren’t any issues with it. The documentation is in need of some work; your best bet at the moment is a combination of the rather terse --help message and reading the source. There are also some hidden gems in the Github commits; for example, details on using .stignore files, a very handy feature, are buried in an earlier commit to README.md that didn’t survive the move of the documentation to the forum.

The only other significant issue I have encountered is that getting all of the nodes across the cluster to connect to each other at the same time seems to be inexplicably difficult. It may just be my setup, but more often than not, nodes (on the LAN) will timeout while trying to connect to at least one of their peers. This isn’t a huge issue as you only really need one chained connection to bring all the data up to date, but it does cause a something of a minor OCD flare up.

Given the early stages of the project, though, and the fact that it is open source, these anomalies are trivial. Overall, Syncthing is a delight. It does exactly what I need it to do, and it does it securely. Given the promise it is showing this early, I can only imagine it getting a lot better over time as more bugs are squashed and features rolled out.

Notes

1. Naturally enough, someone is on the case… Update: 17/5/14 App is available on Google Play and it works pretty well.
2. I added a debian daemon script to a thread on the Syncthing boards as there is no built in daemon functionality (yet).