Code in the hole

How to sync a MySQL table between two remote databases.

nospam@example.com (David Winterbottom) — Fri, 03 Sep 2010 21:04:58 +0000

Definitely tricker than you might think.

Seems like it should be trivial using "SELECT ... INTO OUTFILE" and "LOAD DATA INFILE ..." to make the transfer via dumping the table into a temporary file. However, the "SELECT ... INTO OUTFILE" creates a file on the remote server, rather than locally. This prevents the use of LOAD DATA INFILE for the second step as the file being loaded has to be local, or on the destination server.

Following the guidance in the docs, you can create local dump of a table by using the "--execute" option to output the results of a "SELECT ..." statement into a local file.

mysql -D database_name -e "SELECT ... " > /path/to/file.txt

This works but has two downsides:

First, running a shell command forces you to step outside the MySQL adapter of your progamming language which means it is a new place where the database credentials need to be passed. Shelling out commands always feels like you've failed.
Further, as far as I can tell, you can't control the field separator or line endings using this technique (in the same way as you can with "SELECT ... INTO OUTFILE ...") and so the file includes an unwanted line with the field names and tab-separates the fields.

It's worth noting the mysqldump isn't much help here, as the "--tab" option that allows CSV output to be generated only works with a local database connection.

Now that we've got our data locally, we load it into the remote database using "LOAD DATA INFILE" and make use of the "LOCAL" keyword which lets us use a local data file:

mysql -h x.x.x.x -u user -D database_name --password=... -e \
    "LOAD DATA LOCAL INFILE '/path/to/file.txt' \
     REPLACE INTO TABLE table_name \
     IGNORE 1 LINES"

Of course, you may want to truncate the table first if you want a clean sync. As this operations locks the destination table, it often makes sense to load the data into a temporary copy of the table, and then perform a "RENAME TABLE" operation to swap in the new table.

Here's a quick and dirty PHP implementation:

$tableName = 'some_table';
$sql =
   "SELECT * 
    FROM $tableName";
$pathToCsv = '/tmp/some-file.csv';
$command = sprintf("mysql -h %s -u %s  --password=%s -D %s -e '%s' > %s",
    '10.0.0.2', 
    'db-user', 
    'db-password', 
    'database_name', 
    $sql, 
    $pathToCsv);
exec($command);

$sql =
   "LOAD DATA LOCAL INFILE '$pathToCsv'
    REPLACE INTO TABLE `$tableName`
    CHARACTER SET 'utf8'
    IGNORE 1 LINES";
$db->execute($sql); // Using your favourite database adapter

Phing trick for avoiding deploying debug code

nospam@example.com (David Winterbottom) — Sun, 22 Aug 2010 15:41:54 +0000

Fool me once, shame on you; fool me twice, shame on me

Ensuring mistakes aren't repeated is a commonplace activity for any development team. This can manifest itself in many ways such as writing regression tests, stepping up your code reviews, adding stories to a testing plan or humiliating the developer in question through use of an unusual (dunce's) hat.

We had an issue recently where some debugging code got committed and wasn't picked up during testing. Naturally, this code was picked up once it hit the production environment, we put rapidly put a patch in place. An embarrassing moment but the kind of thing that happens from time to time down in the trenches.

One possible remedy for this would be to add a new sniff to the in-house coding standard (which is tested using the PEAR code sniffer on a SVN pre-commit hook) to look for debugging code and block the commit if any is found. However, a quicker solution was to modify our deployment scripts to search the codebase and bail out if any debug code was found. We use phing as our deployment tool - here's the appropriate section from our build file:

<echo msg="Checking codebase for use of var_dump()" />
<exec command="[ `find ${folder.app.export}/classes -name '*.php' | xargs grep -nr '\(^\s*|\s\+\)var_dump(.*\?);' | wc -l` -eq 0 ]" dir="." checkreturn="true" />

The command here is grepping all PHP files within the SVN export for occurrences of var_dump. If any such matching lines are found then a non-zero exit will be returned and the build will fail. We haven't had such an issue since.

Return false with prudence

nospam@example.com (David Winterbottom) — Thu, 28 Jan 2010 22:00:00 +0000

From "Javascript: the good parts":

It is rarely possible for standards comittees to remove imperfections from a language because doing so would cause the breakdage of all of the bad programs that depend on those bad parts. They are usually powerless to do anything except heap more features on top of the existing pile of imperfections.

Douglas Crockford's terse yet lucid javascript primer makes some excellent points on writing in a language with more than its fair of share of shortcomings. The advice is manyfold: constituting functionality or design decisions to avoid (the "bad" and "awful" parts) as well as patterns and practices making use of the strongest parts of the language. Essentially, one is guided to programming within a subset of the language, avoiding the poor quality and outright dangerous components. This chimes in well with the notation of "programming into a language" rather than within in it - stressed by the seminal "Code Complete" by Steve McConnell.

Don't limit your programming thinking only to the concepts that are supported automatically by your language. The best programmers think of what they want to do, and then they assess how to accomplish their objectives with the programming tools at their disposal.

Both these concepts are relevant to programmers of PHP - a language carrying just as much baggage as javascript. In my experience, developers who have only known PHP are prone to employing of a variety of bad practices and anti-patterns. Such things are fostered by several influences, including the forgiving nature of the language, the veritable wealth of bad advice within the comments on the PHP manual, and a general lack of understanding of the art of object-oriented programming. Indeed, I think it's essential that PHP developers learn to program (and hence think) in other languages: python and java in particular. Reading the work of Martin Fowler is a good place to start but that's a topic for another time.

Continue reading "Return false with prudence"

Javascript refactoring for customising shared libraries

nospam@example.com (David Winterbottom) — Tue, 13 Oct 2009 22:48:42 +0000

One difficulty working with a shared in-house framework is it is difficult to maintain a common javascript file that is valid across multiple applications. This is currently an issue we face at Tangent, where we run a generic e-commerce platform which we customise to the needs of each client. Most of these e-commerce applications have a javascript-rich checkout page whose functionality differs in small yet significant ways such as the required and optional fields within a delivery address, or the range of delivery options available. Ideally, you would have a single javascript file to work for all these checkouts, but these differences make such a file impracticle due to the changes in DOM structure and business logic between applications.

For instance, one problem that arose recently was a requirement to remove the javascript messaging from within certain parts of the checkout for one appliation. The shared checkout.js file features a method which reloads the order totals within the page after a new delivery option has been chosen and displays a message to notify the user:

// [checkout.js]
var shop = {
    checkout: {
        reloadOrderTotals: function() {
            // Do the reloading...
            shop.display.notify("Order totals recalculating...");
        }
}

Unfortunately, another client required the same functionality to reload the order totals but without the notification call. One option would have been to create a local version of the checkout.js file and remove the offending line, but this would have created a larger maintenance overhead going forward due to the large amount of duplicate code.

A neater option would have been to dynamically override the shop.checkout.reloadOrderTotals method in an application-specific file:

// [application.js]
shop.checkout.reloadOrderTotals = function() {
    // Reload order totals but without a notification call
}

However, again this leads to most of the the body of the shop.checkout.reloadOrderTotals method being duplicated just to remove one line.

Instead, drawing inspiration from the excellent Working with Legacy Code by Michael Flowers, a neater solution is to perform a dynamic-language version of the Extract method refactoring to isolate the functionality that we wanted to override into it's own method. Hence, we alter checkout.js to split the reloading and notification functionality into two methods:

// [checkout.js]
var shop = {
    checkout: {
        reloadOrderTotals: function() {
             // Do the reloading...
            shop.checkout.displayOrderTotalNotification("Order totals recalculating...");
        },
        displayOrderTotalNotification(message) {
            shop.display.notify(message);
        } 
}

Now in the local project javascript file, we can null out the messaging behaviour:

// [application.js]
shop.checkout.displayOrderTotalNotification = function(){}

This neat trick allows us to customise our core javascript functionality in a way that does not lead to code duplication. The only thing required to make this work is to ensure that the shared checkout file is loaded before the application-specific one.

A pseudo-code job advert and its discontents

nospam@example.com (David Winterbottom) — Thu, 09 Jul 2009 21:13:00 +0000

Based on the success of a highly tongue-in-cheek ad for a project manager, we recently experimented with a similar approach for finding developers: a job ad written in PHP. Now I appreciate this is deeply lame, but the results of the campaign were quite surprising - more of which in a minute. First, hold your nose and parse the following:

<?php
class TangentLabs extends HoxtonWebCompany implements 
    InnovativeWebsites, WorldBeatingApplications, IngeniousECommerce 
{
    const vacancyForBrilliantDevelopers = true;
    public $benefits = array(
        'Smartest web agency in London',
        'Working on inventive web apps, using cutting-edge technology',
        'Super-friendly work environment, working within genuinely brilliant dev team',
    );
    public $drawbacks = null;
    public $sampleProjects = array(
        'http://www.borders.co.uk',
        'http://www.labour.co.uk',
        'http://www.sap.com/education'
    );
    public static function offerJob(Developer $you) {
        return (self::hasTheSkills($you) && self::hasWowFactor($you));
    }
    private static function hasTheSkills($developer) {    
        $desiredSkills = array('Object-oriented PHP 5', 'Advanced MySQL', 'Flex/AS3'); 
        return (
            count(array_intersect($developer->skills, $desiredSkills)) > 1 &&
            (int)$developer->loveForCoding > 1 << 30
        );
    }
    private static function hasWowFactor($developer) {
        return ($developer instanceof creativeThinker &&
            sizeof($developer->brain) > floatval(strpad('1', 100, '0')) &&
            property_exists($developer, 'hungerForNewTechnologies'));
    }
}
$you = new Developer;
if (true === TangentLabs::offerJob($you)) {
    throw new Party();
}

It's actually quite a fun challenge to write a decent job ad in PHP, conveying both the requirements and writing PHP code that isn't hilariously convoluted. I'd tried to do the same in Bash later on when we were hiring a sys-admin, but gave up after 5 minutes.

As you would expect with such a self-consciously smug job advert like this, torrents of abuse came tumbling down? Salutations along the lines of "You pretentious Shoreditch wankers..." and so forth. However, what was a little unexpected were emails along the lines of.

Hi, I really like your job ad - the best I've seen for ages. I live in California though and don't actually want to apply for a job.

How strange - who can be bothered to type up a complimentary cover letter for a job they don't even want to apply for - most genuine applicants can't be bothered to get the spelling and grammar right in their opening salvos.

Another consequence is that most applicant's responses are also written in PHP pseudocode, which become a bit tiresome after the twentieth implementation of Developer - we were asking for it though.

Strangely though, the clinching factor for several candidates was the following comment, left at the end of the ad:

// If you have questions, just ask, and if you’re a recruitment consultant don’t even 
// think about emailing – our board won’t let us use you, even though you may have our 
// literal exact perfect candidate just waiting for an interview. Sorry.

There's nothing like the mutual hatred of recruitment consultants to bring people together.

Deploying cron jobs using Phing

nospam@example.com (David Winterbottom) — Sun, 31 May 2009 17:32:02 +0000

Deploying applications that depend on cron-jobs can be a pain. However, Phing can be used to make such deployments easy - here's how...

Consider an application folder structure as follows:

/builds
    /development
    /test
    /stage
/src
    /cron.d
        appname-__BUILD__-order-processing
    /scripts
        /order-processing
            handle-ready-to-ship-orders.php
            handle-cancellations.php
            ...
    /public
    /classes
        ...

All development work takes place within the /src folder while the /builds/* folders are used as targets in deployment. This system allows multiple builds to happily co-exist on the same server and the whole application infrastructure to be moved between servers easily as the structure in source control mirrors that of the server.

In this example e-commerce app, we have a number of scripts (in /src/scripts) that handle order-processing which need to be called periodically by the cron daemon. A naive approach in deployment would be to export the codebase to an appropriate build folder but then edit the server's crontab by hand and add the appropriate lines to run these scripts.

This isn't such a great idea though as it relies on that most unreliable facility, human memory, to ensure the build is fully deployed - this creates an unnecessary overhead which is bound to lead to mistakes. Furthermore, the overhead acts as a deterrent for using asynchronous jobs within an application, limiting the app in terms of what it can do. As far as I am aware, there is no easy way to automatically update a user's crontab automatically.

A much better way is to create a number of scripts which specify the cron tasks and are deployed to the /etc/cron.d folder. In the above example, these are stored in /src/cron.d and would look something like:

15 * * * *   root   /var/www/ecommerce.com/builds/__BUILD__/scripts/handle-ready-to-ship-orders.php > /dev/null 2>> /var/log/cron.errors.log
35 * * * *   root   /var/www/ecommerce.com/builds/__BUILD__/scripts/handle-cancellations.php > /dev/null 2>> /var/log/cron.errors.log

(Note that, for some reason, a blank line is required at the end of this file in order for the script to be run by cron.) Here, the __BUILD__ is a tokenised parameter which will be replaced during deployment to configure the path to the appropriate script - a phing trick I've described previously.

One further complication is that if both the dev, test and stage builds are running on the same server, then the scripts deployed to /etc/cron.d could clobber each other as they have the same name within each build. This can be neatly side-stepped using the Phing's glob mapper to replace the __BUILD__ component of the file to be the appropriate build name (similar to how the paths are configured within the file itself).

This is probably best illustrated with a sample phing script which takes the build name (eg "development") as a parameter:

<xml version="1.0" encoding="UTF-8"?>
<project>
    <target name="deploy">
        ...
        <delete>
            <fileset dir="/etc/cron.d/">
                <include name="appname-${build.name}-*" />
            </fileset>
        </delete>
        <copy todir="/etc/cron.d/"> 
            <filterchain>
                <replacetokens begintoken="__" endtoken="__">
                    <token key="BUILD" value="${build.name}" />
                </replacetokens>
            </filterchain>
            <mapper type="glob" from="appname-__BUILD__-*" to="appname-${build.name}-*" />
            <fileset dir="${path.to.build}/cron.d">
                <include name="appname-__BUILD__-*" />
            </fileset>
        </copy>
    </target>
</project>

The illustrated snippet does two things:

Deletes any previous scripts for this build from /etc/cron.d (this is why we namespace the files with "appname" to prevent accidentally removing a system file).
Copies the new scripts into /etc/cron.d while replacing the token __BUILD__ with the build name in both the file contents and file names.

After separate deployments for dev and stage, we should find the following:

/etc
    /cron.d
        appname-development-order-processing
        appname-stage-order-processing
        ...

where, for instance, the contents of /etc/cron.d/appname-development-order-processing would be:

* * * * *   root   /var/www/ecommerce.com/builds/development/scripts/handle-ready-to-ship-orders.php
* * * * *   root   /var/www/ecommerce.com/builds/development/scripts/handle-cancellations.php
...

Having automatic and reliable deployment of cron-jobs in place is quite liberating. Suddenly, lots of application processing can be done asynchronously without worrying about the overhead of maintaining the appropriate crontabs by hand.

Auto-generating an FAQ with Prototype

nospam@example.com (David Winterbottom) — Mon, 25 May 2009 20:34:01 +0000

Have just been writing an FAQ page for commandlinefu.com. Documenting is always tiresome, but FAQs particularly so when hand-coding the HTML links between each question and the summary table at the top of the page.

Javascript to the rescue: I cobbled together a quick Prototype script which automatically generates the FAQ summary links by parsing the DOM for the appropriate links:

document.observe('dom:loaded', function(){
    $$('a.question').each(function(ele){
        var id = ele.innerHTML.unescapeHTML().gsub(/[^\w- ]/, '').gsub(/[\s-]+/, '-').toLowerCase();
        ele.writeAttribute({id: id});
        var link = new Element('a', {href: '#'+id})
            .update(ele.innerHTML)
            .observe('click', function(event){
                event.stop();
                Effect.ScrollTo(ele); 
            });
        $('questions').insert(new Element('li').insert(link));
    });
});

Now, all I have to do is write my FAQ ensuring the questions live in anchor tags with the class "question" and there is an empty list tag at the top of the page to house the summary block.

Here's two snapshots the relevant section of the DOM before and after the dom:loaded event has fired. Before:

<ul id="questions"></ul>
<dl>
    <dt><a class="question">What is this site?</a></dt>
    <dd>...</dd>
    <dt><a class="question">Who created this site?</a></dt>
    <dd>...</dd>
</dl>

and after:

<ul id="questions">
    <li><a href="#what-is-this-site">What is this site?</a></li>
    <li><a href="#who-created-this-site">Who created this site?</a></li>
</ul>
<dl>
    <dt><a class="question" id="what-is-this-site">What is this site?</a></dt>
    <dd>...</dd>
    <dt><a class="question" id="who-created-this-site">Who created this site?</a></dt>
    <dd>...</dd>
</dl>

As you can see, the javascript simply extracts the relevant content from the questions and inserts the appropriate identities and links into the DOM to form the quick-links. Using the Effect.ScrollTo function also gives a pleasant user experience.

Inspecting Javascript objects

nospam@example.com (David Winterbottom) — Sun, 24 May 2009 12:04:00 +0000

Learning Ruby or Python from the command-line prompt is greatly enhanced by the built-in inspection methods these languages provide. These allow the methods and properties of an object to be interrogated via a simple method call which returns an array of all property or method names.

For instance, in IRB (the interactive Ruby command-line) we can interrogate the integer object:

irb(main):001:0>my_int = 1
irb(main):002:0>my_int.methods

This returns an array of all method names for integer objects:

["%", "odd?", "inspect", "prec_i", "<<", "tap", "div", "&", "clone", ">>", "public_methods", "_send_", "object_id", "instance_variable_defined?", "equal?", "freeze", "to_sym", "*", "ord", "+", "extend", "next", "send", "round", "methods", "prec_f", "-", "even?", "singleton_method_added", "divmod", "hash", "/", "integer?", "downto", "dup", "to_enum", "instance_variables", "|", "eql?", "size", "instance_eval", "truncate", "~", "id", "to_i", "singleton_methods", "modulo", "taint", "zero?", "times", "instance_variable_get", "frozen?", "enum_for", "display", "instance_of?", "^", "method", "to_a", "+@", "-@", "quo", "instance_exec", "type", "**", "upto", "to_f", "<", "step", "protected_methods", "<=>", "between?", "==", "remainder", ">", "===", "to_int", "nonzero?", "pred", "instance_variable_set", "coerce", "respond_to?", "kind_of?", "floor", "succ", ">=", "prec", "to_s", "<=", "fdiv", "class", "private_methods", "=~", "tainted?", "_id_", "abs", "untaint", "nil?", "chr", "id2name", "is_a?", "ceil", "[]"]

Meanwhile, in Python we use the build-in function dir():

>>> my_dict = {}
>>> dir(my_dict)

to get a list of attributes (including methods) for a dictionary variable:

['_class_', '_cmp_', '_contains_', '_delattr_', '_delitem_', '_doc_', '_eq_', '_ge_', '_getattribute_', '_getitem_', '_gt_', '_hash_', '_init_', '_iter_', '_le_', '_len_', '_lt_', '_ne_', '_new_', '_reduce_', '_reduce_ex_', '_repr_', '_setattr_', '_setitem_', '_str_', 'clear', 'copy', 'fromkeys', 'get', 'has_key', 'items', 'iteritems', 'iterkeys', 'itervalues', 'keys', 'pop', 'popitem', 'setdefault', 'update', 'values']

Javascript doesn't provide such a feature as standard but it's straightforward to implement a simple inspection object:

var Inspect = {
    TYPE_FUNCTION: 'function',
    // Returns an array of (the names of) all methods
    methods: function(obj) {
        var testObj = obj || self;
        var methods = [];
        for (prop in testObj) {
            if (typeof testObj[prop] == Inspect.TYPE_FUNCTION && typeof Inspect[prop] != Inspect.TYPE_FUNCTION) {
                methods.push(prop);
            }
        }
        return methods;
    },
    // Returns an array of (the names of) all properties
    properties: function(obj) {
        var testObj = obj || self;
        var properties = [];
        for (prop in testObj) {
            if (typeof testObj[prop] != Inspect.TYPE_FUNCTION && typeof Inspect[prop] != Inspect.TYPE_FUNCTION) {
                properties.push(prop);
            }
        }
        return properties;
    }
}

Now we can simply pass in any object (or call without an argument to test the window object) to retrieve an array or properties or methods. For example, working at the Firebug console, we can examine the hash object of the Prototype Javascript library, and the native document.location object:

Moreover, it can sometimes be useful to have a debug mode for an application where these methods are added to the supertype "object" which provides the prototype for all others. In this instance, debug mode is switched on by added a 'debug_mode' parameter to the GET parameters. For convenience, I'm also using the methodize function provided by Prototype:

if (/[?&]debug_mode(&.*)?$/.test(document.location.search)) {
    Object.prototype.properties = Inspect.properties.methodize();
    Object.prototype.methods = Inspect.methods.methodize();
}

Then you can inspect an object without refering to the Inspect object (namespace really):

var my_hash = new Hash();
my_hash.methods();
my_hash.properties()

Code and associated JSunit test suite available at my github repo.

Phing, Xinc and Nabaztags

nospam@example.com (David Winterbottom) — Wed, 06 May 2009 21:57:13 +0000

Finally got around to setting up continuous integration for some of the projects that comprise the day-job. We're using the PEAR package Xinc, which has proved to be excellent thus far - especially as it integrates so well with my deployment tool of choice: Phing. Part of the fun in setting it up was looking for suitable feedback mechanisms or devices. Email notifications are a given but there are a range of more interesting feedback mechanisms available such as toolbar notifications, remote-controlled lava lamps, or plain humiliation tactics (such as making the person who broke the build wear the dunce's hat till it is fixed). By a strange twist of fate there happened to be an unused Nabaztag in the office: a strange Rabbit-like fellow able to play sounds, move its ears and activate a selection of brightly colours LEDs contained in its body. Nabaztags are controlled through a simple HTTP web-service, with the various actions been specified by GET requests using a simple API.

Out of the box, Xinc only provides email notifications for feedback but its design is such that plugins are easy to create. Indeed, Raphael Stolt has written recently on creating a publisher that uses the Growl notifications native to Macs. A natural extension to this would be to create an Linux version, given the inclusion of libnotify in the latest Ubuntu release. That's not for today though.

The true flexibility in Xinc comes from the phingpublisher publisher, which allows an arbitrary target to be called from a phing script. Rather than create a Xinc plugin for the Nabaztag, writing a Phing task seemed to offer more flexibility as it could then be used in deployment scripts elsewhere.

I've previously created a couple of simple phing tasks for updating a Twitter status, and interacting with the Unfuddle API. Creating a Nabaztag task was just a simple extension of these cURL-based tasks. The various attributes mirror those from the API docs.

NabaztagTask.php attributes
Name	Type	Description	Default	Required
serialNum	String	Serial number	n/a	Yes
token	Integer	Token number	n/a	Yes
leftEarPosition	String	Position of the left ear (0-16)	n/a	No
rightEarPosition	String	Position of the right ear (0-16)	n/a	No
message	String	Message.	n/a	No
messageId	Integer	Message id.	n/a	No
voice	Integer	Voice to use (use the API to fetch the full list)	n/a	No
choreography	String	A string which prescribes the choreography to use (see API docs)	n/a	No
choreographyTitle	Boolean	Choreography title	n/a	No
urlList	Boolean	List of URLs to pass to the Nabaztag (can be used for playing audio files)	n/a	No
checkReturn	Boolean	Whether to check the return code of the request, throws a BuildException if the update fails	false	No

The code for this task can be found in my github repo - to install locally, export the file to a local phing tasks folder (I use /usr/share/php/phing/tasks/my on my Ubuntu machine). Having done that, a simple example build.xml file to exercise the task would be:

<?xml version="1.0" ?>
<project name="Example Nabaztag update" basedir=".">
    <taskdef name="nabaztag" classname="phing.tasks.my.NabaztagTask" />
    <target name="build-failure">
        <nabaztag serialNum="${nabaztag.serialNum}" token="${nabaztag.token}" message="The build failed!" voice="US-Liberty" />
    </target>
</project>

To use this with Xinc, you simply need to called the appropriate target from your project configuration file. Another toy example:

<?xml version="1.0"?>
<xinc>
    <project name="Toy example">
        <configuration>
		<setting name="loglevel" value="1"/>
		<setting name="timezone" value="Europe/London"/>
	</configuration>
	<property name="dir" value="/var/xinc/projects/sample"/>
    	<cron timer="*/15 * * * *"/>
        <modificationset>
            <svn directory="${dir}" update="true" />
        </modificationset>		
        <builders>
        	<phingBuilder buildfile="${dir}/build.xml" target="build-project"/>
        </builders>
        <publishers>
            <onfailure>
	        <phingpublisher buildfile="${dir}/build.xml" target="build-failure" />
            </onfailure>
        </publishers>
    </project>
</xinc>

Some of the options for controlling the Nabaztag are only available with the V2 version. The one on my desk is V1 and so I haven't tested every action, I've just followed the instructions in the API docs. Unfortunately, the Nabaztag webservice isn't as RESTful as would be desired - it returns a 200 response code for every request, whether it fails or not. This makes it a touch tricky to handle failed updates.

Nabaztags are also useful for general reminders - I have the following line in my crontab:

15 18 * * 5 curl "http://api.nabaztag.com/vl/FR/api.jsp?sn=$SERIAL&token=$TOKEN&tts=It+is+now+time+to+go+to+the+pub"

Ingenious use of an anonymous function

nospam@example.com (David Winterbottom) — Wed, 15 Apr 2009 23:48:04 +0000

Design a function f, such that: f(f(n)) == -n where n is a 32-bit signed integer; you can't use complex numbers arithmetic.

Just stumbled across the above gem of a question whilst idly browsing Stack Overflow.

Interesting in its own right, what makes this particularly intriguing is that the question doesn't specify a language to use - indeed, the choice of language has a major say in the range of solutions available. The following solution in Python is stunningly elegant:

def f(x):
   if isinstance(x, int):
      return (lambda: -x)
   else:
      return x()

The use of a lambda function is very clever but does feel like cheating a little. Nevertheless, such a solution was a sharp reminder of the dangers of being too versed in a particular language (PHP in my case) such that it's hard to think outside the language constraints. Of course, the above solution can be implemented in any language that supports anonymous functions.

The less clever but more "natural" solution (at least to a mathematician) is for f to toggle the parity of n, multiplying by -1 only for one parity (the cases of positive and negative n need handling separately). It's fairly easy to build up the solution by considering in sequence n=0,1,2,3,… - see the following clunky PHP implementation:

function f($n)
{
    if (0 == $n) return 0;
    if ($n > 0) {
        return ($n % 2 == 1) ? $n + 1 : 1 - $n;
    } else {
        return ($n % 2 == 1) ? $n - 1 : -($n + 1);
    }
}

See the whole range of answers at Stack Overflow for considerably more information.

Using a Phing filter to flush browser caches

nospam@example.com (David Winterbottom) — Sun, 15 Mar 2009 19:01:13 +0000

A quick Phing tip that's made my life easier when deploying new versions of commandlinefu.com.

One of the key performance recommendations from Steve Souders' excellent "High Performance Websites" is to use Expires HTTP headers to set far-future expiration dates for your site components (such as images, Javascript files and CSS stylesheets). This way, browsers can cache the files between requests giving a performance boost to your site. Assuming you're using Apache for serving, the following settings can be used to set these headers for all Javascript and CSS files (there are a few alternative ways of achieving the same result):

<FilesMatch "\.(ico|pdf|flv|jpg|jpeg|png|gif|js|css|swf)$">
    Header set Expires "Thu, 15 Apr 2010 20:00:00 GMT"
</FilesMatch>

The main issue to be aware of using this technique is that, when your components change, you need to ensure your visitors are forced to download the latest version rather than using the one cached by their browser. The only way to ensure this happens is to use a different URL for the assets in question. One option might be to rename the files themselves but a more convenient alternative is to include a query string as part of the request URL (eg <script src="/js/behaviour.js?2009-03-15" type="text/javascript" />). Then changing the query string component is sufficient to force browsers to make a full request for the new component.

This works well but is an easy-to-forget overhead for deployment. However, this substitution can be automated by making use of the Filters that Phing provides. Doing so is trivial: simply insert a tokenised string as the query string after your asset URLs. That is:

<link rel="stylesheet" href="http://codeinthehole.com/css/styles.css?~~CACHEBUSTER~~" type="text/css" />

where the '~~' delimiter indicates the token. Then include something like the following snippet in your Phing deployment script.

<tstamp>
    <format property="build.datetimestring" pattern="%Y-%m-%d-%H-%M" />
</tstamp>
...
<target name="create-temp-build" description="Creates a temporary copy of the source files">	
    <echo msg="Copying deployment files into temporary directory" />
    <copy todir="${dev.folder.temp}">
        <filterchain>
            <replacetokens begintoken="~~" endtoken="~~">
                <token key="CACHEBUSTER" value="${build.datetimestring}" />
            </replacetokens>
        </filterchain>
	<fileset refid="deployment-files" />
    </copy>
</target>

The filterchain component of the copy task parses the given fileset for matching tokens that match and replaces them with the given value. In this example, I'm using timestamps as the replacements as these will ensure a different query string on each deployment. Doing so ensures that the deployed HTML includes the lines:

<link rel="stylesheet" href="http://codeinthehole.com/css/styles.css?2008-03-15-21-51" type="text/css" />
<script type="text/javascript" src="http://codeinthehole.com/js/site-behaviour.js?2008-03-15-21-51">

which then ensure that all subsequent visitors download the latest versions of the CSS and javascript files. The above target is taken from a deployment script I use which creates a temporary snapshot of the codebase that I want to deploy, but the basic principle of using the replacetokens filter is easily transferable to any deployment script.

The most important command-line tip - incremental history searching with .inputrc

nospam@example.com (David Winterbottom) — Tue, 03 Feb 2009 23:40:00 +0000

Getting www.commandlinefu.com off the ground has renewed my interest in Bash, UNIX and all things command-line. Powerful one-liners are things of beauty and are worth collecting; however what I consider to be the most influential command-line tip I know covers four:

"\e[A": history-search-backward
"\e[B": history-search-forward
"\e[C": forward-char
"\e[D": backward-char

These lines need to be placed in your ~/.inputrc file, the start-up script for the Readline utility used by Bash (as well as several other applications) and others). The important commands here are the first two, which bind your up and down cursor keys to incrementally search your history. (The second two ensure that left and right continue to work correctly).

This is incredibly useful for retrieving commands you've used previously and makes a huge difference to your productivity. For instance, to find a previous SSH command from a few days ago, simply type "ss" and press up a few times. This will allow you to browse through all your previous ssh… commands until you find the right one - you never need to use more than 4 or 5 keystrokes to retrieve any previous command. If your cycling through too many commands to find the right one, type in a few more characters to refine the search.

As indicated above, this functionality is available in all applications that use Readline including MySQL, Python, IRB (interactive Ruby shell) and others. Once you're used to this feature, it's hard to live without - the first thing I do once I've been set up as a user on a new server is update my .inputrc file to contain these settings. The one place where I sorely wish this functionality existed is the Firebug Javascript commandline in Firefox (a ticket already exists requesting a similar feature).

Another way of searching your history is to use CTRL+R, which essentially performs a full-text search on your history (keep pressing CTRL+R to cycle through results). In this case, searching for "ssh" will locate all commands that feature this string anywhere in the command. Although this is actually a more powerful feature than the incremental history search described above, I don't often use it as: (a) the incremental search generally lets me jump to the desired command in fewer key-presses and (b) I find "ssh", CTRL+R, CTRL+R slightly awkward to type and less intuitive than "ssh", UP, UP. Horses for courses really - you could probably be just as efficient with either one.

» Hat-tip to the place where I first learnt this: Power Shell Usage by Simon Myers

One extra thing: this functionality can be neatly complimented by some choice history settings in your ~/.bashrc file:

export HISTSIZE=1000000
export HISTFILESIZE=1000000000

These simply set your history to be very large so that you have a huge bank of commands to search.

Current pet project: Command-Line-Fu

nospam@example.com (David Winterbottom) — Thu, 22 Jan 2009 23:16:22 +0000

If you're anything like me, you spend a lot of time at the UNIX command-line manipulating the filesystem, configuring Linux, playing with services and so forth. As any UNIX user knows, tremendous power can be wielded through judicious function selection, piping and output redirection. It's often quite staggering what can be achieved in a single line given a rudimentary knowledge of sed, grep, awk, cut…

Indeed, when I stumble upon a line of particular elegance or usefulness, I generally log them to a Tomboy note (fired up in a flash using Gnome-do). This has proved extremely useful as I am often returning to the list to recall how to, say, rsync a fileset with an exclude list - it's generally faster than Googling or going to the man pages. As time has progressed though, this small repository has grown into a sizable collection and finding a particular command quickly is now a problem. Motivated by this problem and taking inspiration from my daily reading (Hacker News, Programming Reddit and Stack Overflow), I'm in the process of putting together a lightweight web-app for cataloguing and ranking notable UNIX one liners.

The basic idea is for users to be able to store their useful one liners on the site for (a) retrieval in the future and (b) sharing with others who undoubtedly will need to save the same problem. Once created, these commands are parsed for the functions used and any relevant tags to provide good navigational props. The individual commands also be discussed and commented on. Further, users can vote on each others commands allowing simple leaderboards to be constructed. After a few months, it will be interesting to see what the top 10 most useful awk commands are.

It's only been a week (working piecemeal after dinner) but the site's nearly ready for a beta release. It's constructed using a combination of CodeIgniter and Zend Framework, making use of the fast and lightweight nature of CodeIgniter coupled with the extensive range of components that ZF provides. The site's called http://www.commandlinefu.com and should be live in about a week assuming I find some spare time over the weekend. Watch this space.

Phing task to create an Unfuddle message

nospam@example.com (David Winterbottom) — Sun, 11 Jan 2009 21:50:48 +0000

Another day, another new Phing task; again integrating with project management software - this time the excellent Unfuddle.

I've been playing with Unfuddle for a few days now and it's very impressive. You get SVN and git hosting as well as superb issue tracking. It also supports simple project messages (which are displayed on the project dashboard) and so-called notebooks which are essentially project wikis that can be used to house documentation and manuals. One great feature of integrated project management software is the ability to merge news from a variety of sources (SVN commits, changes in ticket status, changes to notebooks) onto a single page that provides a snapshot of the latest activity on a project. Unfuddle does this on each project dashboard, where the latest messages are displayed along side news of the latest SVN and issue activity. The task detailed below provides a means for Phing to automatically add information to this dashboard page by creating a new message.

This extension is very similar to my previous Phing task (for updating a Twitter status), making use of the cURL library to POST XML to Unfuddle. In this case though, the Unfuddle API for creating a message offers a few extra options such as categorising your messages. The task supports the following attributes:

UnfuddleMessageTask.php attributes
Name	Type	Description	Default	Required
subdomain	String	Subdomain of Unfuddle account (eg. 'example' from http://example.unfuddle.com).	n/a	Yes
projectid	Integer	Project id (eg. 123 from http://example.unfuddle.com/projects/123/).	n/a	Yes
username	String	Username.	n/a	Yes
password	String	Password.	n/a	Yes
title	String	Message title.	n/a	Yes
body	String	Message body.	''	No
categoryid	Integer	The category id of the message.	''	No
categoryids	String	A comma-separated list of category ids (eg. 1,2,3).	''	No
checkreturn	Boolean	Whether to check the return code of the request, throws a BuildException the update files.	false	No

The only thing to note here is that you can choose whether you specify a single category id or a collection - it wouldn't make sense to specify both these attributes.

An example build.xml using this task would be:

<?xml version="1.0" ?>
<project name="Example Unfuddle update" basedir="." default="message">
<tstamp>
<format property="build.time" pattern="%Y-%m-%d %H:%I" />
</tstamp>
<taskdef name="unfuddlemessage" classname="phing.tasks.my.UnfuddleMessageTask" />
<target name="message">
<unfuddlemessage
subdomain="example"
projectid="12345"
username="exampleuser"
password="password"
title="Deploying to live site at ${build.time}"
body=""
categoryid="4" />
</target>
</project>

This simply creates a new Unfuddle message with the time of the last build. This is an overly simplified example - see my previous post for a sample parameterised deployment target that would allow a dynamic message to be created by different targets within the deployment file.

The source code for TwitterUpdateTask.php is as follows (with docblocks stripped out for brevity):

<?php
require_once "phing/Task.php";
class UnfuddleMessageTask extends Task
{
const URL_TEMPLATE_UPDATE = 'http://%s.unfuddle.com/api/v1/projects/%d/messages';

// Twitter response codes
const HTTP_RESPONSE_OK = 200;
const HTTP_RESPONSE_CREATED = 201;
const HTTP_RESPONSE_BAD_REQUEST = 400;
const HTTP_RESPONSE_BAD_CREDENTIALS = 401;
const HTTP_RESPONSE_BAD_URL = 404;
const HTTP_RESPONSE_METHOD_NOT_ALLOWED = 405;
const HTTP_RESPONSE_SERVER_ERROR = 500;
const HTTP_RESPONSE_BAD_GATEWAY = 502;
const HTTP_RESPONSE_SERVICE_UNAVAILABLE = 503;

private static $responseMessages = array(
self::HTTP_RESPONSE_BAD_REQUEST => 'Bad request - you may have exceeded the rate limit',
self::HTTP_RESPONSE_BAD_CREDENTIALS => 'Your username and password did not authenticate',
self::HTTP_RESPONSE_BAD_URL => 'The Unfuddle URL is invalid',
self::HTTP_RESPONSE_METHOD_NOT_ALLOWED => 'The specified HTTP verb is not allowed',
self::HTTP_RESPONSE_SERVER_ERROR => 'There is a problem with the Unfuddle server',
self::HTTP_RESPONSE_BAD_GATEWAY => 'Unfuddle is either down or being upgraded',
self::HTTP_RESPONSE_SERVICE_UNAVAILABLE => 'Unfuddle servers are refusing request',
);

private $subdomain;
private $projectId;
private $username;
private $password;
private $title;
private $body;
private $categoryIds;
private $checkReturn = false;

public function setSubdomain($subdomain)
{
$this->subdomain = $subdomain;
}
public function setProjectId($projectId)
{
$this->projectId = (int)$projectId;
}
public function setUsername($username)
{
$this->username = $username;
}
public function setPassword($password)
{
$this->password = $password;
}
public function setTitle($title)
{
$this->title = $title;
}
public function setBody($body)
{
$this->body = $body;
}
public function setCategoryId($categoryId)
{
$this->categoryIds = array((int)$categoryId);
}
public function setCategoryIds($categoryIdList)
{
$this->categoryIds = explode(",", $categoryIdList);
}
public function setCheckReturn($checkReturn)
{
$this->checkReturn = (boolean)$checkReturn;
}

public function init()
{
if (!extension_loaded('curl')) {
throw new BuildException("Cannot update Unfuddle", "The cURL extension is not installed");
}
}
public function main()
{
$this->validateProperties();

$curlHandle = curl_init();
curl_setopt($curlHandle, CURLOPT_URL, $this->getUpdateUrl());
curl_setopt($curlHandle, CURLOPT_USERPWD, "$this->username:$this->password");
curl_setopt($curlHandle, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curlHandle, CURLOPT_HTTPHEADER, array('Accept: application/xml', 'Content-type: application/xml'));
curl_setopt($curlHandle, CURLOPT_POST, true);
curl_setopt($curlHandle, CURLOPT_POSTFIELDS, $this->getRequestBodyXml());
$responseData = curl_exec($curlHandle);
$responseCode = curl_getinfo($curlHandle, CURLINFO_HTTP_CODE);
$errorCode = curl_errno($curlHandle);
$errorMessage = curl_error($curlHandle);
curl_close($curlHandle);

if (0 != $errorCode) {
throw new BuildException("cURL error ($errorCode): $errorMessage");
}
$this->handleResponseCode((int)$responseCode);
}
private function validateProperties()
{
if (!$this->subdomain) {
throw new BuildException("You must specify a subdomain");
}
if (!$this->projectId) {
throw new BuildException("You must specify a project id");
}
if (!$this->username || !$this->password) {
throw new BuildException("You must specify an Unfuddle username and password");
}
if (!$this->title) {
throw new BuildException("You must specify a message title");
}
}
private function getUpdateUrl()
{
return sprintf(self::URL_TEMPLATE_UPDATE, $this->subdomain, $this->projectId);
}
private function getRequestBodyXml()
{
$xmlWriter = new XMLWriter();
$xmlWriter->openMemory();
$xmlWriter->startElement('message');
$xmlWriter->writeElement('title', $this->title);
$xmlWriter->writeElement('body', $this->body);

if ($this->categoryIds) {
$xmlWriter->startElement('categories');
foreach ($this->categoryIds as $categoryId) {
$xmlWriter->startElement('category');
$xmlWriter->writeAttribute('id', "$categoryId");
$xmlWriter->endElement();
}
$xmlWriter->endElement();
}
$xmlWriter->endElement();
return $xmlWriter->flush();
}
private function handleResponseCode($code)
{
if ($code == self::HTTP_RESPONSE_CREATED) {
$this->log("New Unfuddle message posted: '$this->title'", Project::MSG_INFO);
return;
}
if (array_key_exists($code, self::$responseMessages)) {
$this->handleFailedUpdate(self::$responseMessages[$code]);
} else {
$this->handleFailedUpdate("Unrecognised HTTP response code '$code' from Unfuddle");
}
}
private function handleFailedUpdate($failureMessage)
{
if (true === $this->checkReturn) {
throw new BuildException($failureMessage);
}
$this->log("New Unfuddle message unsuccessful: $failureMessage", Project::MSG_WARN);
}
}

The fully documented source and associated example build.xml file are available to download:

» UnfuddleMessageTask.zip (2.6kb)

Phing task to update Twitter status

nospam@example.com (David Winterbottom) — Sat, 10 Jan 2009 21:57:00 +0000

At Tangent Labs, we're currently experimenting with integrating Twitter into our project workflow to provide a latest activity feed in a easily digestible format (for both developers and non-technical people). For a pilot project, we've created a Twitter account and added an SVN post-commit hook script that updates Twitter with the latest commit information (commit message, affected files, author). We're going to integrate our bug-tracking software shortly too but that's not the subject of this post.

Instead, I'm going to detail a custom Phing task I've written that updates the project Twitter account. This allows notices of builds (to test, stage and production) to be integrated into a single feed. One of the great things about Twitter is its API and the range of applications already written to interact with it. My current favourite client is Gwibber, which (amongst other things) displays a small pop-up whenever the account gets a new update. Having this running while working on the project is great for staying informed with the latest activity, be it new commits, opened tickets or deployments.

The task I've written is TwitterUpdateTask.php and should be copied into your $PATH_TO_PHING/ext/my/ folder (create it if it doesn't exist already). Mirroring the format of the Phing docs, this task has the following attributes:

UpdateTwitterTask.php attributes
Name	Type	Description	Default	Required
username	String	Twitter username	n/a	Yes
password	String	Twitter password	n/a	Yes
message	String	Update message	n/a	Yes
checkreturn	Boolean	Whether to check the return code of the request, throws a BuildException the update files.	false	No

A simple example build.xml file using this task is as follows:

<?xml version="1.0" ?>
<project name="Simple Twitter update" basedir="." default="tweet">
<tstamp>
<format property="build.time" pattern="%Y-%m-%d %H:%I" />
</tstamp>
<taskdef name="twitterupdate" classname="phing.tasks.my.TwitterUpdateTask" />
<target name="tweet">
<twitterupdate
username="example" password="mypassword"
message="Build at ${build.time}" />
</target>
</project>

This simply updates the Twitter status with the time of the last build. A more useful means of using this task is to parameterise the Twitter target to take a specified message so that it can be called from different deployment targets:

<?xml version="1.0" ?>
<project name="Example Twitter update" basedir="." default="deploy-to-test">
<tstamp>
<format property="build.time" pattern="%Y-%m-%d %H:%I" />
</tstamp>
<taskdef name="twitterupdate" classname="phing.tasks.my.TwitterUpdateTask" />
<target name="tweet">
<twitterupdate
username="dave_test" password="eggnog"
message="${twitter.status}" />
</target>
<target name="deploy-to-test">
<phingcall target="tweet">
<property
name="twitter.status"
value="Deploying to test: ${build.time}" />
</phingcall>
</target>
<target name="deploy-to-stage">
<phingcall target="tweet">
<property
name="twitter.status"
value="Deploying to stage: ${build.time}" />
</phingcall>
</target>
<target name="deploy-to-production">
<phingcall target="tweet">
<property
name="twitter.status"
value="Deploying to production: ${build.time}" />
</phingcall>
</target>
</project>

There are lots of extensions from this idea such as updating Twitter with continuous integration results, failed builds, code coverage metrics.

The source code for TwitterUpdateTask.php is as follows (with docblocks stripped out for brevity):

<?php
require_once "phing/Task.php";
class TwitterUpdateTask extends Task
{
const URL_TEMPLATE_UPDATE = 'http://twitter.com/statuses/update.xml?status=%s';
const MAXIMUM_MESSAGE_LENGTH = 140;

// Twitter response codes
const HTTP_RESPONSE_SUCCESS = 200;
const HTTP_RESPONSE_NOT_MODIFIED = 304;
const HTTP_RESPONSE_BAD_REQUEST = 400;
const HTTP_RESPONSE_BAD_CREDENTIALS = 401;
const HTTP_RESPONSE_FORBIDDEN = 403;
const HTTP_RESPONSE_BAD_URL = 404;
const HTTP_RESPONSE_SERVER_ERROR = 500;
const HTTP_RESPONSE_BAD_GATEWAY = 502;
const HTTP_RESPONSE_SERVICE_UNAVAILABLE = 503;

private static $responseMessages = array(
self::HTTP_RESPONSE_NOT_MODIFIED => 'Status hasn\'t changed since last update',
self::HTTP_RESPONSE_BAD_REQUEST => 'Bad request - you may have exceeded the rate limit',
self::HTTP_RESPONSE_BAD_CREDENTIALS => 'Your username and password did not authenticate',
self::HTTP_RESPONSE_FORBIDDEN => 'Forbidden request - Twitter are refusing to honour the request',
self::HTTP_RESPONSE_BAD_URL => 'The Twitter URL is invalid',
self::HTTP_RESPONSE_SERVER_ERROR => 'There is a problem with the Twitter server',
self::HTTP_RESPONSE_BAD_GATEWAY => 'Twitter is either down or being upgraded',
self::HTTP_RESPONSE_SERVICE_UNAVAILABLE => 'Twitter servers are overloaded and refusing request',
);

private $username;
private $password;
private $message;
private $checkReturn = false;

public function setUsername($username) {
$this->username = $username;
}
public function setPassword($password) {
$this->password = $password;
}
public function setMessage($message)
{
$this->message = trim($message);
}
public function setCheckReturn($checkReturn)
{
$this->checkReturn = (boolean)$checkReturn;
}
public function init()
{
if (!extension_loaded('curl')) {
throw new BuildException("Cannot update Twitter", "The cURL extension is not installed");
}
}
public function main()
{
$this->validateProperties();
$curlHandle = curl_init();
curl_setopt($curlHandle, CURLOPT_POST, true);
curl_setopt($curlHandle, CURLOPT_POSTFIELDS, array());
curl_setopt($curlHandle, CURLOPT_URL, $this->getUpdateUrl());
curl_setopt($curlHandle, CURLOPT_USERPWD, "$this->username:$this->password");
curl_setopt($curlHandle, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curlHandle, CURLOPT_HTTPHEADER, array('Expect:'));
$twitterData = curl_exec($curlHandle);
$responseCode = curl_getinfo($curlHandle, CURLINFO_HTTP_CODE);
$errorCode = curl_errno($curlHandle);
$errorMessage = curl_error($curlHandle);
curl_close($curlHandle);
if (0 != $errorCode) {
throw new BuildException("cURL error ($errorCode): $errorMessage");
}
$this->handleTwitterResponseCode((int)$responseCode);
}
private function validateProperties()
{
if (!$this->username || !$this->password) {
throw new BuildException("You must specify a Twitter username and password");
}
if (!$this->message) {
throw new BuildException("You must specify a message");
} elseif (strlen($this->message) > self::MAXIMUM_MESSAGE_LENGTH) {
$this->message = substr($this->message, 0, self::MAXIMUM_MESSAGE_LENGTH);
$this->log("Message is greater than the maximum message length - truncating...", Project::MSG_WARN);
}
}
private function getUpdateUrl()
{
return sprintf(self::URL_TEMPLATE_UPDATE, $this->getEncodedMessage());
}
private function getEncodedMessage()
{
return urlencode(stripslashes(urldecode($this->message)));
}
private function handleTwitterResponseCode($code)
{
if ($code == self::HTTP_RESPONSE_SUCCESS) {
$this->log("Twitter status updated to: '$this->message'", Project::MSG_INFO);
return;
}
if (array_key_exists($code, self::$responseMessages)) {
$this->handleFailedUpdate(self::$responseMessages[$code]);
} else {
$this->handleFailedUpdate("Unrecognised HTTP response code '$code' from Twitter");
}
}
private function handleFailedUpdate($failureMessage)
{
if (true === $this->checkReturn) {
throw new BuildException($failureMessage);
}
$this->log("Update unsuccessful: $failureMessage", Project::MSG_WARN);
}
}

The fully documented source and associated example build.xml file are available to download:

» TwitterUpdateTask.zip (2.6kb)