As the title suggests, I expect this to be the first in a series of articles about how I use org-mode. Right now I’m at the point where I’m really starting to see the light. I’ve known about org-mode for several years, but only recently decided to give it a serious try.

Since I started my own company, I have been looking for a simple, effective tool for keeping track of all the various projects I am working on. I’m already using Harvest to track billable hours. For detailed planning of each sprint in an ongoing project, I love Pivotal Tracker, and I’m going to stick with that for the time being.

But I needed something else to get an overview of all those projects and ideas that are floating around in various stages of completion. Both in my head, in emails, on my hard drive and in Google Docs. Something to record all sorts of metadata about each project (estimates, ideas, contracts) as well as actual work artifacts (pointers to code commits, design docs, links to user stories).

I’m going to skip the details, but I briefly evaluated Highrise (for projects and tasks), but quickly discarded it. I may explain why in a separate post, but the gist of it is just that the workflow didn’t really feel natural to me.

So I decided to have another go at org-mode, and this time I think I nailed it!

Basics

Org-mode is an Emacs major mode which provides features for organizing information in a hierarchical manner. Lists of items can be nested into sublevels, and each item can contain plain text contents. There is support for links to URLs and to places in other files.

There is also support for lots and lots of other stuff. A much better introduction can be found at the Org-mode website.

File structure

One of the things I found hard to begin with was file organization. How do you layout the directory structure? Should you create a folder per project, with files like “plan.org”, “estimates.org” etc. inside each? Or maybe organize the files by contact?

Eventually, I ended up simply having a bunch of files in a single directory:

buzz.org 
leads.org 
links.org 
notes.org 
passwords.org 
projects.org 
readinglist.org 
todo.org 
wb.org

Side note 1: passwords.org is of course encrypted and, incidentally, replaces my use of 1Password (which worked fine but I never bothered to learn 90% of its features so the UI felt very bloated to me).

Side note 2: The file notes.org collects stuff I capture with remember.el.

Everything related to projects (whether active, planned or potential), goes into projects.org. Each project just gets a top-level item with related information organized into sub-items.

Artifacts

This was the real revelation for me: Links.

It sounds trivial, and it is, but linking to all kinds of stuff seems to be one of the best ways for me to organize my work. I have no idea why I didn’t realize this sooner. One of the reasons I tried out Highrise was because of its support for organizing emails. I still think that this is pretty clever, but org-mode is just simpler and more versatile.

Example:


The text “Data model” links to the Google Doc containing the data model description. Likewise, the text “Mail about data model” links to an email (in Gmail) I sent to the client about the data model.

This is a very simple building block, but very powerful. I can insert into my project any resource I can link to, which means almost anything these days. Commits in Github, web pages (obviously), documents, spreadsheets, presentations, video clips.

Estimates

Org-mode is ideal for breaking down major tasks into subtasks, so it makes sense that it also has support for attaching estimates to these tasks.

What you see here is column mode in action. The estimates on the lowest level tasks (cyan items) are entered by hand, and Org-mode then sums up these values in the higher level items, ending up at the root with a total estimate of 36 hours.

Tasks

Highrise has support for tasks, and I’m sure it’s good.

However, I created support for tasks just by creating the file todo.org. I really don’t need anything more fancy than a list of text strings right now. When I do, org-mode has lots of nice features for tracking deadlines, time spent on tasks etc.

Exporting to PDF

This is really easy once you have everything set up.

Suppose I want to export this very simple file:

This is done with the command org-export-as-pdf, and here is the resulting PDF.

Next steps

I’m not sure, what do you think? Any suggestions for improvements on these workflows would be greatly appreciated. Just drop a comment.

If you’re OK with dark text on a light background, you can stop reading now.

Personally, I have always preferred a dark background when coding, so obviously my Emacs theme is a dark one. And switching back and forth between Emacs and Terminal.app is much easier on the eyes if the background in Terminal is dark too.

Unfortunately, the Terminal theme I like best (“Pro”) doesn’t agree with ANSI colors. Especially the color blue becomes hard to read:

Many people resolve to using iTerm instead, but I’m not really a fan of it. The shortcuts for switching between tabs are on the arrow keys, much slower than the Terminal shortcuts. And maybe I’m missing something, but I never managed to reconfigure those shortcuts. To be honest, I think the iTerm configuration UI is a bit of a disaster.

So, I discovered by accident that I can actually tweak the background color in Terminal to make the blue color readable.



This is the color picker for the background color. The vertical slider will adjust the brightness of the background, right? Well, it turns out that the brightness of the colored text is also adjusted, but in intervals. The blue color becomes brighter to begin with, then falls back to the original color at about 10% on the slider. The same happens for the other colors, but at different points on the slider. For example the red color switches at about 20% and the magenta a bit after that.

 

 

 

 

 

 

What works best for me is setting the slider at about 9%, just before the blue color falls back.

It may be hard to see from the pictures above, but this makes a huge difference, at least on my monitor. Reading the colored text is no longer a strain on the eye.

Using Gmail efficiently

January 7, 2011

Most of us get a lot of email.

If you’re using Gmail, here are a few simple tips on how to get a better overview of your crowded inbox. These all work well for me, but feel free to pick and choose, they’re not related.

  1. Archive anything you won’t need again anytime soon
  2. Star items that you need to respond to
  3. Use labels and filters to help finding messages visually when looking through your inbox

Tips in action

 

Archiving

When you archive a message (shortcut: e), it disappears from your inbox, but can still be found under “All mail” (shortcut: g a).

If you fail to do this immediately, you’ll soon end up with a lot of crap littering your inbox. To clean up, move the cursor between messages (shortcuts: j and k), select each message to be archived (shortcut: x), then archive all selected messages (shortcut: e).

Starring

If you read a message that you need to respond to, do so right away or “star” it (shortcut: s) to make it appear in the Starred folder (shortcut: g s). When you have some time to spare, just check this folder and be sure to remove the star (shortcut: s) on each mail after responding to it.

Using labels

Set up labels for a few broad categories of mails you get. Don’t be too specific – labels in 15 different colors throughout you mailbox are not helpful.

Mails can be labelled manually, either when reading the mail (shortcut: l + the first few letters of the label you want) or in the overview by selecting them first. This is tiresome and you’ll probably end up forgetting from time to time.

Much better: Have Gmail label them automatically when received. Under “Settings -> Filters”, you can setup multiple filters that match incoming mail by sender’s mail address (full or just domain), subject, content keywords etc.

By the way, an undocumented feature here is that you can use OR to specify multiple values. So this is a valid filter:

Matches: from:(foo.com OR bar.com OR steve@apple.com)
Do this: Apply label "Working Bits"

And by the way…

Did you notice that I mentioned the keyboard shortcut of each command above? This is actually the important stuff, the tips were just to get your attention.

OK, not really, but using the shortcuts in Gmail is by far the biggest single productivity gain I have found so far. I never use the mouse anymore – everything I need to do can be done with shortcuts, and it is much, much faster.

JRuby/Clojure integration

December 21, 2010

I recently did a demo at an aarhus.rb meeting, showing how to call Clojure code from a Rails app. Specifically, I wanted to try using the very impressive Incanter library which is written in Clojure. This is not just a contrived example – Incanter is something you’d want to integrate with if you needed to do something statistics related.

This post is a walkthrough of how I got a very simple demo app running. It displays a histogram and allows the user to enter the size of the normal distribution sample.

NOTE: The finished Rails app can be found on Github.

1. Set up the basics

As with any other project, I am using RVM for managing multiple (J)Ruby implementations and keeping gemsets project-specific.

So first, create a directory for the Rails app. Inside that directory, create a .rvmrc file containing the following line:

rvm --create jruby-1.5.2@jruby-clojure-demo

Now do a “cd .” to have RVM pick up this file. This switches to jruby-1.5.2 (which must be installed, see RVM docs) and a newly created “jruby-clojure-demo” gemset.

We obviously need Rails for this demo:

gem install rails

This installs Rails 3.0.x as of this writing. Since this takes a little while to complete, fire up another terminal and let’s move on.

2. Fetch all the JARs we need with Leiningen

Leiningen is a build/dependency management tool for Clojure. I’m not using it to build anything for this demo, but it is useful for pulling down all the JARs we need.

Install Leningen, then create a new project with “lein new”, eg. in ~/tmp. Modify the project.clj file to look something like this:

(defproject incanter-test "1.0.0-SNAPSHOT"
  :description "FIXME: write"
  :repositories {"incanter" "http://repo.incanter.org"}
  :dependencies [[org.clojure/clojure "1.1.0"]
                 [org.clojure/clojure-contrib "1.1.0"]
                 [org.incanter/incanter-full "1.0.0"]]
  :dev-dependencies [[swank-clojure "1.2.1"]])

Finally run “lein deps” to have Leiningen download all required JAR files into lib/. This also takes a while, but now our Rails gem is installed and we can move on.

3. Create the Rails app

We use -O to skip activerecord:

rails new . -O

Next, add jrclj to the Gemfile. This is a JRuby/Clojure bridge created by Kyle Burton, and it’s the heart of this integration.

gem 'jrclj'

Do a “bundle install” to install all gems required by our new Rails project.

4. Enabling Clojure integration in the Rails project

To allow us to call Clojure code from the Rails project, we need to load all the JARs we downloaded in step 2.

First, create a “deps” folder inside the Rails app and copy all the JARs into it. Then, insert the following before the Bundler#require call in application.rb:

require 'java' 
Dir["#{File.dirname(__FILE__)}/../deps/*.jar"].each do |jar|
  require jar
end

This code needs to be executed before the Bundler call because the imported JARs are needed for the JRClj bridge to initialize properly.

5. Creating the histogram view

It’s time to call some Clojure code to render our view. I’m not showing how to create the actual Rails controller, the view or the simple form that collects the size attribute. See the Github repo for a working example.

To create the histogram, we use the method incanter.charts/histogram. The input to this method is the result from incanter.stats/sample-normal, and we save its output with incanter.core/save.

The output of incanter.core/save is a stream of PNG data. It should have been possible to simply save this data into a new ByteArrayOutputStream (yes, with JRuby you can just instantiate that class), but because of what looks like a glitch in the JRuby/Clojure bridge, this did not work. It looks like the return types may be wrong in some cases, but I haven’t found time to investigate yet.

So I resorted to a dirty hack: Dropping the PNG data into a file in public/images and simply inserting an image_tag into the view to render it. Thread-safe? I don’t think so 🙂

    clj = JRClj.new
    %w{core stats charts}.each { |l| clj._import("incanter.#{l}") }
    clj.eval clj.read_string "(incanter.core/save \
                                (incanter.charts/histogram \
                                   (incanter.stats/sample-normal #{@size.to_i})) \
                                 \"#{::Rails.root.to_s}/public/images/out.png\")"

So what I’m doing here shows one of the two ways you can use JRClj: Evaluating a string containing Clojure code. The clj.eval call above evaluates to the value of the call to incanter.core/save, ie. the outermost sexp in the string.

The other, arguably cleaner and more Ruby-like mode to go, is to instantiate a JRClj object (like above), then call the Clojure functions as if they were Ruby function, with the difference that dashes in method names are replaced with underscores:

jruby-1.5.2 > clj = JRClj.new
=> (truncated...)
jruby-1.5.2 > clj._import("incanter.stats")
=> (truncated...)
jruby-1.5.2 > clj.sample_normal(1000).take(5)  
=> [-1.67885083887929, 0.332869879174951, -0.178234653916897, -0.382876843067349, 1.16325259920326]

Appendix: A Clojure-enabled IRB

I think most Ruby programmers agree that the IRB is a very useful thing, and it becomes even more useful when playing with stuff like JRClj.

Included in the JRClj documentation is a small script that loads the necessary JAR files and fires up an IRB session. I used this while working on the demo, and the script can be found as “jrepl” in the root of the Github repo.

MoinMoin encoding issue

September 4, 2010

As mentioned in my previous post, the encoding in my MoinMoin wiki was somehow broken. When entering Danish characters on a page, they were turned into garbage (which persisted when switching to edit view). Surprisingly, I fixed this by accident today when making a change to improve the performance of the wiki.

It turns out that the preferred way to run a MoinMoin instance on Apache2 these days is with mod_wsgi. I’m not even sure what that is, except it has to do with running Python code in an Apache server via some daemons.

To install mod_wsgi:

sudo aptitude install libapache2-mod-wsgi

The only change needed in my Apache2 configuration was replacing this line:

ScriptAlias /w "/usr/share/moin/server/moin.cgi"  

with these lines:

WSGIScriptAlias /w /var/www/chopwiki/moin.wsgi

WSGIDaemonProcess chopwiki-daemon user=www-data group=www-data home=/var/www processes=5 threads=10 maximum-requests=1000 umask=0007

WSGIProcessGroup chopwiki-daemon

 

The “moin.wsgi” referenced above was copied from /usr/share/moin/server into /var/www/chopwiki but not modified. 

After making this change, the character encoding issue was gone as well! I have no idea why, and I’m not going to spend time finding out.

It seems like I’m going to be haunted by my disastrous server upgrade for some time to come.

The server has gitosis installed, which is what I use for serving my private Git repos. Now, when trying to clone a repo (from another computer), I suddenly got a stack trace:

/usr/bin/gitosis-serve:5: UserWarning: Unbuilt egg for setuptools [unknown version] (/usr/lib/python2.6/dist-packages)
  from pkg_resources import load_entry_point
Traceback (most recent call last):
  File "/usr/bin/gitosis-serve", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 2655, in <module>
    working_set.require(__requires__)
  File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 648, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 546, in resolve
    raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: gitosis==0.2
fatal: The remote end hung up unexpectedly

It turns out that this was because Python had been upgraded to 2.6. Fixing it was extremely easy: Re-install gitosis with the current version of Python:

git clone git://eagain.net/gitosis

cd gitosis

sudo python setup.py install

Now I just need to figure out how to fix the charset encoding in my MoinMoin wiki…

Thinking like a sysadmin

August 24, 2010

I’ll admit right away: Sitting under our staircase, hunched over my laptop, is not really my idea of a great weekend.

I spent a significant part of last weekend in deep frustration. My home server had crashed, and it was absolutely my own fault. When this server is down, it means no internet connection in the house. Plus, a mailing list, my wiki, redirection to my blog and a few other services are down. So obviously this needed to be fixed, and fast.

The trouble began when I tried to create a RAID array last week. After some hassle with a cheap SATA controller “with Raid”, I decided that I would just go for a software RAID. Now, in an attempt to be responsible and actually prepare this operation, I googled a bit and found various posts about how to go about doing this in Ubuntu, caveats to avoid and so on. Among other interesting facts, I discovered that the kernel I was running was known to cause some trouble with RAIDs.

So time to ugrade to the newest release, a leap of 3 or 4 Ubuntu releases. Unfortunately, my patience was running low at this point, so I ignored the HUGE warning and started the upgrade over my SSH connection. Because I couldn’t be bothered to pull out the server from under the staircase and attach keyboard and monitor to it.

As you will have guessed by now, this was an enormous mistake. One of the stupidest things I have yet done in my career as an amateur sysadmin. Because of course, somewhere in the course of the upgrade, something went wrong and broke the network connection. Along with DNS, DHCP, internet connectivity and everything else we rely on this machine for.

After a reboot, I was still unable to contact the machine. So I had to pull it out and attach an actual monitor to it. To my horror, it turned out that the boot process died instantly, because it was unable to mount the root filesystem. WTF?

Let me just cut to the chase here, because too many hours were spent trying out solutions that just didn’t work.

The solution was slightly interesting, though. Not having an internet connection (except on my HTC Hero) made it pretty hard to google for possible solutions, look up GRUB docs etc., so I ended up taking a different approach. I had an Ubuntu 10.4 installer CD lying around, so I did a fresh install using some free space on the hard drive. Then, I compared the GRUB configurations of the broken and the fresh installations. It turned out that the upgrade had been interrupted before it had created an initrd for the new kernel! So I was a bit surprised that it had updated the GRUB config to a half-baked state – the entry in menu.lst contained a kernel line but no initrd.

So stealing a fresh kernel+initrd from the new installation and Voila!, my system was back to normal (except for lots of unconfigured packages etc.).

Lesson learned: As a software developer, I’m enthusiastic, optimistic and sometimes downright reckless. As a sysadmin, pretty much the same. A good sysadmin needs to be conservative and cautious, and I understand why they sometimes get annoyed with us developers.

Keep this in mind the next time your local sysadmin doesn’t “just update that shit” right away.