TL;DR

The backup gem contains a bug that causes it to crash when the HOME environment variable is not set. Notably, this is the case when jobs are started by ‘god’.

I have patched this and will soon send a pull request to the authors of the gem.

Read on to get a few hints on how to track down issues like this one.

The story of a painful bug hunt

0. Background

On several projects I’m using god to start and monitor daemon processes, especially resque workers. On one particular project, this just stopped working recently and as a workaround I had to start the resque workers manually under tmux.

Today I finally took the time to figure out what was going on.

1. Symptoms

In my god.conf I have a ‘resque’ watch configured which starts five workers. Here is the relevant snippet from god.conf:

num_workers = rails_env == 'production' ? 5 : 2
num_workers.times do |num|
  God.watch do |w|
    w.name     = "resque-#{num}"
    w.group    = 'resque'
    w.interval = 60.seconds
    w.env      = {"QUEUE"=>"*", "RAILS_ENV"=>rails_env}
    w.start    = "bundle exec bootup_rake -f #{rails_root}/Rakefile environment resque:work"
    w.dir      = "#{rails_root}"
...

When doing a “god start resque”, god tries to start up the workers but they crash almost immediately. If I do “ps aux | grep resque” within a few seconds I see the processes but then they just disappear.

2. What is god saying?

Finding the scroll that collects the utterings of god is easy:

/var/log/god.log

However, it quicly becomes apparent that god has nothing interesting to say:

I [2011-11-15 16:24:57]  INFO: Using pid file directory: /var/run/god
I [2011-11-15 16:30:55]  INFO: Syslog enabled.
I [2011-11-15 16:30:55]  INFO: Using pid file directory: /var/run/god
I [2011-11-15 16:39:30]  INFO: Syslog enabled.
I [2011-11-15 16:39:30]  INFO: Using pid file directory: /var/run/god
...

A bit of googling suggests that I try to run god in the foreground to see what is going on.

3. Non-daemonized god

So let’s see how god is being invoked:

root@foo:~# ps aux | grep god
root     27956  4.5  0.2 179304 19132 pts/1    Sl   10:12   \
   0:00 /usr/local/rvm/rubies/ruby-1.9.2-p290/bin/ruby      \
   /usr/local/rvm/gems/ruby-1.9.2-p290/bin/god -P /var/run/god.pid -l /var/log/god.log

We need to make a few tweaks:

  • don’t send output to a log file (remove -l)
  • load the config file with -c <config>
  • keep it in the foreground with -D

So our command becomes:

root@foo:~# /usr/local/rvm/rubies/ruby-1.9.2-p290/bin/ruby \ 
   /usr/local/rvm/gems/ruby-1.9.2-p290/bin/god             \
   -P /var/run/god.pid -c /etc/god.conf -D

This yields slightly more interesting output. Here are the lines for the worker ‘resque-1’:

I [2011-11-16 10:17:13]  INFO: resque-1 move 'unmonitored' to 'init'
I [2011-11-16 10:17:13]  INFO: resque-1 moved 'unmonitored' to 'init'
I [2011-11-16 10:17:13]  INFO: resque-1 [trigger] process is not running (ProcessRunning)
I [2011-11-16 10:17:13]  INFO: resque-1 move 'init' to 'start'
I [2011-11-16 10:17:13]  INFO: resque-1 start: bundle exec bootup_rake \ 
   -f /var/www/foo/current/Rakefile environment resque:work
I [2011-11-16 10:17:13]  INFO: resque-1 moved 'init' to 'start'
I [2011-11-16 10:17:13]  INFO: resque-1 [trigger] process is running (ProcessRunning)
I [2011-11-16 10:17:13]  INFO: resque-1 move 'start' to 'up'
I [2011-11-16 10:17:13]  INFO: resque-1 moved 'start' to 'up'
I [2011-11-16 10:17:13]  INFO: resque-1 [ok] memory within bounds [8952kb] (MemoryUsage)
I [2011-11-16 10:17:14]  INFO: resque-1 [ok] process is running (ProcessRunning)

OK, maybe that’s not so interesting after all. God starts the process and is satisfied with that.

What IS interesting is the fact that this actually works! The resque workers keep running and start processing jobs.

4 WTF?

So when I start god non-daemonized, it works. When god is started with ‘service god start’, it does not.

I need a way to see the log output from god when it is running in daemonized mode. This turns out to be as simple as:

god log resque

By default this gives me the output from the first worker in the group. I won’t repeat it here, but it is exactly the same as when running god in the foreground.

It seems to be safe to assume that god is behaving the same way whether started by me or via /etc/init.d/god. To be sure I also tried starting it by executing the exact two lines from this file. Same result.

5. Inspecting the environment

I have suspected for a while that this issue has to do with the environment in which the rake command is being run. This seems to be the only major source of difference between the working and the non-working experiments.

So I need to know the environment in which the failing rake invocations are being run. I know that for all running processes, the environment is available in the file /proc/<pid>/environ. But how can I get to this file when the process dies within seconds?

My solution is to run the following command immediately after god tries to start the workers:

less -f /proc/`ps aux | grep resque | head -1 | cut -d ' ' -f 4`/environ

What this does is

  • ps aux | grep resque finds the process info for the about-to-die resque workers
  • head -1 grabs the first one
  • cut -d ‘ ‘ -f 4 cuts out the PID (fourth field when using a space as the delimiter)
  • This PID is then used as part of the path passed to the outer less command

The environ file is in some kind of binary format (which is why I need -f on the less command) the details of which I haven’t bothered to look up.

I don’t really need to, because after running the above command I see this in my iTerm:

GEM_HOME=/usr/local/rvm/gems/ruby-1.9.2-p290^@TERM=xterm^@IRBRC=/usr/local/rvm/r\
ubies/ruby-1.9.2-p290/.irbrc^@MY_RUBY_HOME=/usr/local/rvm/rubies/ruby-1.9.2-p290\
^@QUEUE=*^@rvm_path=/usr/local/rvm^@PATH=/usr/local/rvm/gems/ruby-1.9.2-p290/bin\
:/usr/local/rvm/gems/ruby-1.9.2-p290@global/bin:/usr/local/rvm/rubies/ruby-1.9.2\
-p290/bin:/usr/local/rvm/bin:/var/www/foo/shared/bundle/ruby/1.9.1/bin:/usr/loca\
l/rvm/gems/ruby-1.9.2-p290/bin:/usr/local/rvm/gems/ruby-1.9.2-p290@global/bin:/u\
sr/local/rvm/rubies/ruby-1.9.2-p290/bin:/usr/local/rvm/bin:/usr/local/sbin:/usr/\
local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games^@PWD=/var/www/foo/releases/20\
111115153743^@LANG=en_US.UTF-8^@SHLVL=1^@BUNDLE_GEMFILE=/var/www/foo/releases/20\
111115153743/Gemfile^@RAILS_ENV=production^@rvm_ruby_string=ruby-1.9.2-p290^@GEM\
_PATH=/usr/local/rvm/gems/ruby-1.9.2-p290:/usr/local/rvm/gems/ruby-1.9.2-p290@gl\
obal^@RUBYOPT=-I/usr/local/rvm/gems/ruby-1.9.2-p290/gems/bundler-1.0.18/lib -rbu\
ndler/setup^@BUNDLE_BIN_PATH=/usr/local/rvm/gems/ruby-1.9.2-p290/gems/bundler-1.\
0.18/bin/bundle^@RUBY_VERSION=ruby-1.9.2-p290^@

Notice that the variables are separated by some byte that less renders as “^@”. Good!

6. Reproducing the bug

Off to Emacs to transform this into something useful. I won’t bore you with the details, but a quick keyboard macro transforms the above into:

export GEM_HOME=/usr/local/rvm/gems/ruby-1.9.2-p290
export TERM=xterm
export IRBRC=/usr/local/rvm/rubies/ruby-1.9.2-p290/.irbrc
export MY_RUBY_HOME=/usr/local/rvm/rubies/ruby-1.9.2-p290
export QUEUE=*
export rvm_path=/usr/local/rvm
export PATH=/usr/local/rvm/gems/ruby-1.9.2-p290/bin:/usr/local/rvm/gems/ruby-1.9.2-p290@global/bin:\
   /usr/local/rvm/rubies/ruby-1.9.2-p290/bin:/usr/local/rvm/bin:/var/www/foo/shared
export PWD=/var/www/foo/releases/20111115153743
export LANG=en_US.UTF-8
export SHLVL=1
export BUNDLE_GEMFILE=/var/www/foo/releases/20111115153743/Gemfile
export RAILS_ENV=production
export rvm_ruby_string=ruby-1.9.2-p290
export GEM_PATH=/usr/local/rvm/gems/ruby-1.9.2-p290:/usr/local/rvm/gems/ruby-1.9.2-p290@global
export RUBYOPT=-I/usr/local/rvm/gems/ruby-1.9.2-p290/gems/bundler-1.0.18/lib -rbundler/setup
export BUNDLE_BIN_PATH=/usr/local/rvm/gems/ruby-1.9.2-p290/gems/bundler-1.0.18/bin/bundle
export RUBY_VERSION=ruby-1.9.2-p2

Back on the server, I clear all existing environment vars with a snippet I found here:

unset  $ ( /usr/bin/env | egrep '^(\w+)=(.*)$' | \
  egrep -vw 'PWD|USER|LANG' | /usr/bin/cut -d= -f1);

Next, I paste all the exports above into iTerm and finally I am ready to try starting the worker again.

Voila!! It crashes:

rake aborted!
can't convert nil into String
/var/www/foo/shared/bundle/ruby/1.9.1/gems/backup-3.0.16/lib/backup.rb:40:in `join'
/var/www/foo/shared/bundle/ruby/1.9.1/gems/backup-3.0.16/lib/backup.rb:40:in `<module:Backup>'
/var/www/foo/shared/bundle/ruby/1.9.1/gems/backup-3.0.16/lib/backup.rb:8:in `<top (required)>'
...

Here are the lines around line 40 of backup.rb:

##                                                                                                                                                                                                                                                                                                                                                                      
# Backup's Environment paths                                                                                                                                                                                                                                                                                                                                            
PATH               = File.join(ENV['HOME'], 'Backup')
DATA_PATH          = File.join(ENV['HOME'], 'Backup', 'data')

Ah, so the backup gem assumes that the HOME environment variable is set. If not, it crashes.

7. Done

Fortunately the source for “backup” is on Github, so I could easily fork the project and apply the very simple patch that was needed.

If you need to run scheduled jobs in your Rails app (!= background jobs as in Resque), you should use the excellent whenever gem.

However, one detail that is really easy to miss is which shell environment the cron job actually runs under.

bundle: command not found

And an easy way to confuse yourself even further is by trying to debug issues in this area by:

  1. Logging into your  server as the deployment user
  2. Doing crontab -l to see which entries whenever installed
  3. Testing an entry by cut’n’pasting it to the shell prompt
The outcome will not be reliable, because you’re running it from a login shell and cron is not. So for example, your rbenv shims may be in the path while testing, but cron does not load your .bashrc or .bash_profile files.

On a project I work on, the staging and production machines are currently quite different. In production we use RVM (globally installed) while staging has rbenv locally installed for the deployment user.

So in staging I need to assume that no binaries are available and use the full path to bundler, in my case /home/passenger/.rbenv/versions/1.9.2-p290/bin/bundle.

(I know, symlinking this to eg. /usr/local/bin/bundle would probably be more elegant, but I actually like the reminder of where my ruby is actually installed. I don’t spend much time on the servers, so I tend to forget.)

Running stuff only on certain stages

EDIT: See Markham’s comment for a better way to do achieve the same result.

Another issue I struggled with for a while is how to schedule jobs only in production. For example, I don’t want to run S3 backups from staging.

I use capistrano/multistage, so I needed to inspect the value of the current “stage” from schedule.rb. It turns out that by including this:

require 'capistrano/ext/multistage'

set :whenever_environment, defer { stage }
require "whenever/capistrano"

…the stage is available in schedule.rb as @environment. Very simple, but it took some googling to find.

Example schedule.rb

Below is my anonymized schedule.rb. It should be pretty self explanatory, post questions in the comments if not.

def production?
 @environment == 'production'
end

set :bundler, production? ? "gp_bundle" : "/home/passenger/.rbenv/versions/1.9.2-p290/bin/bundle"

job_type :gp_rake, "cd :path && RAILS_ENV=:environment :bundler exec rake :task --silent :output"
job_type :gp_runner, "cd :path && :bundler exec rails runner -e :environment ':task' :output"
job_type :gp_bundle_exec, "cd :path && RAILS_ENV=:environment :bundler exec :task"

every 5.minutes do
 gp_rake "thinking_sphinx:index"
end

every 30.minutes do
 gp_runner "FooBar.update"
end

every 1.day, :at => "4:00am" do
 gp_runner "Baz.purge_inactive"
end

if production?
 every 1.day, :at => "4:00am" do
   gp_bundle_exec 'backup perform --trigger my_backup --config-file config/backup.rb --log-path log'
 end
end

EDIT: Jakob Skjerning posted this snippet in the comments:

job_type :rake, “cd /var/www/appname/#{environment}/current && /home/appname/.rvm/gems/ree-1.8.7-2010.02@appname-#{environment}/bin/bundle exec /home/appname/.rvm/wrappers/ree-1.8.7-2010.02@appname-#{environment}/rake –silent RAILS_ENV=#{environment} :task :output”

I’ve long found it annoying that the active tab in Google Chrome has almost the same color as the inactive ones.

This can be easily fixed by installing a custom theme such as this one. The active tab in this theme is dark and the inactive ones light. The contrast makes it possible for me to see in my peripheral vision where I am when switching between tabs.

Don’t know why I didn’t think of this before…

I’m running Gitosis on my home Linux server. I like to keep everything under source control, and this setup allows me to have dozens of small projects with very little overhead.

This is really convenient, except every time I start a new project I have to

  • Add it to gitosis-admin, commit and push
  • Set up the initial Git repo for the project
  • Touch, add and commit a README
  • Setup the remote
  • Do the inital push with “-u”

Obviously, this gets pretty tedious, so I automated it with the following script.

#!/bin/bash

PROJECT=$1
THIS_SCRIPT=`readlink $0`
GITOSIS_ADMIN_DIR=${THIS_SCRIPT%/*/*}
GIT_SERVER=chopmo.dk

echo "Project:" $PROJECT
echo "Gitosis admin is at $GITOSIS_ADMIN_DIR"
echo "Adding project to gitosis-admin"
cd $GITOSIS_ADMIN_DIR
sed -i "" "/just-me/,\$s/writable.*/& $PROJECT/" gitosis.conf
git add gitosis.conf
git ci -m "Adding new project $PROJECT"
git push

echo "Setting up local repo"
mkdir $PROJECT
cd $PROJECT
git init
touch README
git add README 
git ci -m "initial commit"
git remote add origin git@$GIT_SERVER:$PROJECT.git
git push -u origin master

echo "Done."

It is stored under “bin” in my gitosis-admin repo and symlinked to ~/bin/new_project (which is why the readlink at the top is important).

The “sed” call appends the new project name to the list of projects which are writable by only me. The just-me,$ addresses are necessary because the structure of my gitosis.conf is:

[gitosis]

[group gitosis-admin]
writable = gitosis-admin
members = ...

[group just-me]
writable = foo bar baz ...

members = ...

So without the limiting addresses, I would append the new project to the writable projects in the gitosis-admin group as well.

My bash skills are pretty weak, so feedback on the script would be very welcome. But this works, and it means that the amount of time from I get an idea for a new project until I can start hacking on it is brought down to just a few seconds.

 

Xoom review

September 1, 2011

For some reason, I have completely neglected to write about my favourite gadget of the year: The Motorola Xoom.

I just love this device. It is perfect for casual browsing, checking email, checking Twitter, Facebook, newssites and all that waste of time. And Amazon has created an excellent tablet-optimized Kindle app, which is probably what I’ve spent the most hours using (until now that I’ve bought a real Kindle – more about that in an upcoming post).

It has more storage space that I’m going to need anytime soon (32GB I think), it has Wifi of course, excellent battery life, nice and responsive screen. In short, the hardware is great and the build quality feels good.

This is the device that Google uses for developing Honeycomb, meaning that it is always up to date and contains no wierd-ware. This is just stock Android 3.x, and everything works together as it should.

App highlights: Kindle, Google Maps, the builtin browser, BeyondPod HD, Gmail, IMDB, the list goes on. We’re really starting to see some great tablet-optimized apps, even if most of them are from Google so far.

BeyondPod really stands out, they have done an excellent job of adapting their phone version (which I like a lot) to the larger screen. It offers a much better blog reading experience than eg. Google Reader in the browser.

Well, it can’t be all good, can it? On to the bad stuff.

Most obvious flaw: The reflective screen. This sucks exactly as much as it does on a laptop and makes it impossible to use the device outside except at night. I ended up buying a screen protector which has solved this problem at the cost of a little screen clarity. But you get used to that, and the screen protector also eliminates ugly fingerprints on the screen. It even makes it possible to use the tablet even if your fingers aren’t completely dry which was a welcome surprise. It is really a bitch to apply it, but well worth the effort.

Second slight drawback: It’s a bit heavy for extended reading sessions. However, you quickly get used to resting it on something.

Conclusion: If you like Android and want a tablet, I highly recommend this device. It’s more geeky than the iPad, but if you’re a geek like me, you’ll appreciate this.

~/bin/remind

August 29, 2011

Here is a very simple helper script I came up with today:

#!/bin/sh
grep $1 ~/.bash_history | uniq

I’m currently using Maven on a client project and I keep forgetting the various invocations of mvn. So this helps remind me.

Example:

[ ~/wb/foo ] $ remind mvn
mvn 
which mvn
mvn -v
mvn -cpu hpi:create
mvn package
mvn --help
mvn -o package
mvn install
mvn -DdownloadSources=true -DdownloadJavadocs=true -DoutputDirectory=target/eclipse-classes eclipse:eclipse
mvn hpi:run

This is handy for those cases where I want an overview of all ways in which I’ve recently used a command.

If I just want to recall a specific command, I use bash’s Ctrl-R and whichever few letters of the command I remember:

ctrl-R ec

This will recall the long second-to-last command in the example above.

 

I use my Xoom mainly for reading eBooks, and I really like Amazon’s Kindle app.

Unfortunately, it seems to have no support at all for syncing books that are not bought through Amazon (not surprisingly perhaps). This is a problem for me because I often buy from the Pragmatic Programmers, and I don’t want to copy those .mobi files to my tablet manually (I know that PragProg offers to sync the titles via Whispernet, but that only works for physical Kindle devices).

What I found out recently is that there is a pretty easy fix for this:

  1. Store your books in some folder in your Dropbox, say “/kindle”
  2. Install the app FolderSync Lite on your Android device
  3. Set up your Dropbox account in FolderSync. The interface is not very intuitive so pay attention.
  4. Set up a “folderpair” in FolderSync, mapping /kindle in your Dropbox to /mnt/sdcard/kindle on your device. Make sure the sync type is “to local folder”.
  5. Schedule this folderpair to sync as often as you like.
If you got the options in FolderSync right, you can now put your non-Amazon purchases into /kindle in your Dropbox and they will automatically appear in the Kindle app on your Android device.

You may be wondering if this conflicts with the books you bought through Amazon – after all, they’re also stored in /mnt/sdcard/kindle. The trick is that FolderSync does not take ownership of this folder, it simply copies files from the remote (Dropbox) folder into it. So I assume that it doesn’t even clean up deleted files, although I haven’t tried.

A few weeks ago I attended a great Geek night at Trifork. The speaker was @drkrab, and he was talking about “Actor oriented programming”.

Knowing Kresten’s work in recent years, it is obvious that this talk was very inspired by Erlang. In fact, much of it was about Erlang, but he only briefly touched on his Erjang project (which was nice for those of us who saw his Erjang talk at last year’s GOTOcon).

Erlang

Erlang is “designed for reliability in the presence of errors”. Others (notably @drkrab) can explain what this means better than me, but the following is a quick tour of some of the highlights as I understood them from the talk.

An actor is an “active object” (ie. an object containing its own thread) plus a state machine. In Erlang, actors are implemented as processes which are designed to be very lightweight. Kresten mentioned that the size of a process in Erlang is comparable to that of a Hashtable in Java, so literally millions of processes can coexist in the same Erlang VM. Processes can be hooked up to monitor each other, so that if one process dies, it can be restarted by its monitor process. Error handling is done very differently than in other languages: In general, don’t try to cover all possible error cases by coding defensively. Instead, let stuff crash and make sure that the system as a whole is resilient.

Processes communicate with each other by passing messages in the form of immutable data structures. Often, communication will simply be one-way, or it may be indirect in the sense that process A receives a message from process C in response to a message it sent to process B. In this scenario, process B will typically have spawned process C as a worker for handling that particular request. The interface of an actor is called a protocol. It can be thought of as an API except that it may change with the internal state of the actor.

Revolution

The basic premise of Kresten’s talk was that we as developers are headed towards a revolution. Just like object orientation was a revolution about 15 years ago, the internet age will be forcing to change horses again soon.

This is a very interesting theory which sounds very plausible to me. I’ve heard the topic mentioned many times before in different forms, but this was the first time I had heard anyone systematically explore what may be going on right now, and I found that really interesting.

Kresten started by exploring the anatomy of a scientific revolution, using the classic example of the geocentric vs. the heliocentric model of our solar system. Assuming that the earth is at the centre of the universe obviously made the orbits of our neighboring planets look very strange. Also, I’m sure there was a multitude of other problems I know nothing about.

Revolution needed

So there were a lot of observations that were hard to fit into the model, and this is a sign that a revolution is needed. And indeed, a very painful revolution was needed in which lots of scientific effort was suddenly rendered obsolete, and lots of frustration had to be endured.

Software professionals saw the same thing happen with the advent of object orientation. There were lots of misconceptions and resistance initially, but as history has shown, OO turned out to be a very useful paradigm.

However, we may be on the verge of the next revolution. The internet age is profoundly changing the way we build software – everything is now much more decoupled than is used to be. We often build new systems by integrating lots of existing services, each of which is outside our control. So increasingly, we need to design for reliability in the presence of errors. Sounds familiar?

My thoughts

I have thought about this talk a couple of times over the last few weeks, and I am pretty convinced that Kresten is on to something. And when I try to put these ideas in the context of my day to day work, there’s at least a partial match.

Much of what I do is Ruby on Rails programming, and in that community we do a lot of integration work. Not integration in the SOA sense, but integration in the form of consuming services provided by other parties. More and more frequently, we use web services to supply functionality that is just easier to buy than to develop from scratch. A good example is mailing lists. I don’t know anyone who would still opt for a DIY mailing list system when we have services like Mailchimp which are feature-rich, affordable and very solid. It has just become cheaper to buy services like this. Error reporting is another good example. I pay $5 per month for my Hoptoad account – this is so cheap that I wouldn’t dream of rolling my own solution. It just wouldn’t be cost effective because my time is much better spent working for clients.

These external services may be down from time to time, and we need to be prepared for that. Were we to implement them in our own software, they would be down from time to time as well, but that downtime would often coincide with the downtime of the client software (which is especially scary in the case of error reporting – related, check Avdi Grimm’s war story on the latest PragProg podcast).

Apart from RoR I do a lot of Android programming, and some of the ideas apply here as well, but I think putting them into practice is further off into the future. Mobile devices certainly need to handle errors well (loss of network connectivity is a very real issue), but changing the programming paradigm to something like Erlang doesn’t seem to be right around the corner.

However, the Android app I’m working on right now is written in Scala, so maybe I should try to sneak in a few actors 😉

 

One of the things that bothers me the most in software development is when you come across a problem for which there simply is no nice solution to be found. You come up with some ugly hack and get the job done, but the nagging feeling of having fallen short lingers for days.

Problem in question: A project I’m working on sells stuff and uses a payment gateway for this. API access is expensive so we’re opting for the proxy method, meaning that the page that reads credit card information is actually served by the payment provider. It looks something like this:

http://epay.dk/proxy.cgi/http://myshop.com/payment

This means that our /payment page is a very special case. All asset paths must be relative, ie. images/foo.png instead of /images/foo.png. In contrast, links must be absolute, http://shop.com/about instead of /about.

I want the rest of the site to follow normal Rails conventions, so I need to special case this very page. This in itself is a challenge, because as it turns out my ugly hack will affect the application layout as well.

I first tried a few different routes:

  1. Override url_for and set :path_only => false. This solves only half of the problem (asset paths would still be wrong) and turned out to be painful because of the two different url_for implementations. Figuring out why is left as an exercise for the reader because frankly, I can’t be bothered to explain why. Leave a comment if you want to know.
  2. Define some default_url_options. For some reason, I never got this to work reliably, and after some time of fruitless digging trough the Rails source, I abandoned the idea.
  3. Create middleware that kicks in only if controller == Xxx and action == Yyy. It would then process the HTML and change href=”/…” instances to be relative etc. You may object that this is a terrible way of abusing the middleware system, but if it had worked, it would actually have been the least intrusive option. The reason it broke down was unexpected and enlightening: I was able to rewrite the HTML, but afterwards the Apache module pagespeed rewrote it again to optimize CSS includes etc. Obviously this only happened in our production environment, so it took me a couple of hours to track down.
So it would seem that my quest to solve this (somewhat) elegantly had failed. So be it. Time to resort to the nasty hacks.

The solution

Warning: You will be appalled.

What I ended up doing was:

  • In views/layouts/application.html.haml, I only include JS and CSS if we’re not on the payment page.
  • In the payment view, I include all JS/CSS like this (notice that the path does not begin with a slash): %script{ :src => “javascripts/jquery.min.js”, :type => “text/javascript” }
  • Also in the payment view, I added some JS that absolutifies all links. It looks like this:
  $('a[href^="/"]').each(function (idx) {
    fullUrl = "http://myshop.com" + $(this).attr("href");
    $(this).attr("href", fullUrl);
  });

Conclusion

I have created some code that works, but which I’m really, really unhappy about. I hope that some day I’ll come up with a better idea and fix this, but I doubt it. Suggestions are obviously welcome – please leave a comment.

Feeding the trolls

May 31, 2011

Today, after reading Zed Shaws latest blog post, I did something really stupid.

Of all internet phenomena I’ve come in contact with, trolling is probably the worst. And although I think Zed is not helping the situation by being so harsh, I can understand why he is really, really pissed off.

In response to this, I started a repo on Github of known trolls. I tweeted about it and Mentalizer correctly pointed out that my repo was more or less an act of trolling in itself. Such a list is worthless without some kind of public criteria for who goes on there, or at least some sensible definition of “trolling”. I offered neither, and I wasn’t prepared to put in the effort, making my initiative decidedly half-assed.

So my public repo of trolls was a bad idea. What do we need instead?

Another bad idea

Here is another well intended but probably really bad idea. I’ll build a site that allows you to propose trolls and allows registered users to vote up their troll-ranking. It will be complete with an API to retrieve someone’s troll-score and automatically blacklist them from your site.

Why is this a bad idea? Well obviously, if I actually built such a site, I would be the first name on the list, voted up by hundreds of actual trolls. So by my own definition, I’d be a troll. If this became popular (which it would have to in order to fullfill its goal), it would become a warzone of mudslinging and attempts to destroy people’s reputation.

Conclusion

This is not the first time I have thought about how to help the internet getting rid of trolls, but hopefully it will be the last, because I feel like I’m wasting my time. I think it would be a huge accomplishment to reduce the amount of trolling to a tolerable level, on the same scale as what Google is doing to fight spam or what Stack Overflow is doing to fight ignorance.

But it all comes back to the old advice: Don’t feed the trolls. By creating any public list, voting based system or whatever, I would be feeding the trolls. My problem is that I don’t really understand the psychology of trolls, so I keep thinking up all kinds of elaborate systems which all fail in the same way: By acknowledging the trolls, feeding them.

Fighting something by ignoring it is really hard.