Code in the hole (Entries tagged as deployment)

Phing trick for avoiding deploying debug code

nospam@example.com (David Winterbottom) — Sun, 22 Aug 2010 15:41:54 +0000

Fool me once, shame on you; fool me twice, shame on me

Ensuring mistakes aren't repeated is a commonplace activity for any development team. This can manifest itself in many ways such as writing regression tests, stepping up your code reviews, adding stories to a testing plan or humiliating the developer in question through use of an unusual (dunce's) hat.

We had an issue recently where some debugging code got committed and wasn't picked up during testing. Naturally, this code was picked up once it hit the production environment, we put rapidly put a patch in place. An embarrassing moment but the kind of thing that happens from time to time down in the trenches.

One possible remedy for this would be to add a new sniff to the in-house coding standard (which is tested using the PEAR code sniffer on a SVN pre-commit hook) to look for debugging code and block the commit if any is found. However, a quicker solution was to modify our deployment scripts to search the codebase and bail out if any debug code was found. We use phing as our deployment tool - here's the appropriate section from our build file:

<echo msg="Checking codebase for use of var_dump()" />
<exec command="[ `find ${folder.app.export}/classes -name '*.php' | xargs grep -nr '\(^\s*|\s\+\)var_dump(.*\?);' | wc -l` -eq 0 ]" dir="." checkreturn="true" />

The command here is grepping all PHP files within the SVN export for occurrences of var_dump. If any such matching lines are found then a non-zero exit will be returned and the build will fail. We haven't had such an issue since.

Deploying cron jobs using Phing

nospam@example.com (David Winterbottom) — Sun, 31 May 2009 17:32:02 +0000

Deploying applications that depend on cron-jobs can be a pain. However, Phing can be used to make such deployments easy - here's how...

Consider an application folder structure as follows:

/builds
    /development
    /test
    /stage
/src
    /cron.d
        appname-__BUILD__-order-processing
    /scripts
        /order-processing
            handle-ready-to-ship-orders.php
            handle-cancellations.php
            ...
    /public
    /classes
        ...

All development work takes place within the /src folder while the /builds/* folders are used as targets in deployment. This system allows multiple builds to happily co-exist on the same server and the whole application infrastructure to be moved between servers easily as the structure in source control mirrors that of the server.

In this example e-commerce app, we have a number of scripts (in /src/scripts) that handle order-processing which need to be called periodically by the cron daemon. A naive approach in deployment would be to export the codebase to an appropriate build folder but then edit the server's crontab by hand and add the appropriate lines to run these scripts.

This isn't such a great idea though as it relies on that most unreliable facility, human memory, to ensure the build is fully deployed - this creates an unnecessary overhead which is bound to lead to mistakes. Furthermore, the overhead acts as a deterrent for using asynchronous jobs within an application, limiting the app in terms of what it can do. As far as I am aware, there is no easy way to automatically update a user's crontab automatically.

A much better way is to create a number of scripts which specify the cron tasks and are deployed to the /etc/cron.d folder. In the above example, these are stored in /src/cron.d and would look something like:

15 * * * *   root   /var/www/ecommerce.com/builds/__BUILD__/scripts/handle-ready-to-ship-orders.php > /dev/null 2>> /var/log/cron.errors.log
35 * * * *   root   /var/www/ecommerce.com/builds/__BUILD__/scripts/handle-cancellations.php > /dev/null 2>> /var/log/cron.errors.log

(Note that, for some reason, a blank line is required at the end of this file in order for the script to be run by cron.) Here, the __BUILD__ is a tokenised parameter which will be replaced during deployment to configure the path to the appropriate script - a phing trick I've described previously.

One further complication is that if both the dev, test and stage builds are running on the same server, then the scripts deployed to /etc/cron.d could clobber each other as they have the same name within each build. This can be neatly side-stepped using the Phing's glob mapper to replace the __BUILD__ component of the file to be the appropriate build name (similar to how the paths are configured within the file itself).

This is probably best illustrated with a sample phing script which takes the build name (eg "development") as a parameter:

<xml version="1.0" encoding="UTF-8"?>
<project>
    <target name="deploy">
        ...
        <delete>
            <fileset dir="/etc/cron.d/">
                <include name="appname-${build.name}-*" />
            </fileset>
        </delete>
        <copy todir="/etc/cron.d/"> 
            <filterchain>
                <replacetokens begintoken="__" endtoken="__">
                    <token key="BUILD" value="${build.name}" />
                </replacetokens>
            </filterchain>
            <mapper type="glob" from="appname-__BUILD__-*" to="appname-${build.name}-*" />
            <fileset dir="${path.to.build}/cron.d">
                <include name="appname-__BUILD__-*" />
            </fileset>
        </copy>
    </target>
</project>

The illustrated snippet does two things:

Deletes any previous scripts for this build from /etc/cron.d (this is why we namespace the files with "appname" to prevent accidentally removing a system file).
Copies the new scripts into /etc/cron.d while replacing the token __BUILD__ with the build name in both the file contents and file names.

After separate deployments for dev and stage, we should find the following:

/etc
    /cron.d
        appname-development-order-processing
        appname-stage-order-processing
        ...

where, for instance, the contents of /etc/cron.d/appname-development-order-processing would be:

* * * * *   root   /var/www/ecommerce.com/builds/development/scripts/handle-ready-to-ship-orders.php
* * * * *   root   /var/www/ecommerce.com/builds/development/scripts/handle-cancellations.php
...

Having automatic and reliable deployment of cron-jobs in place is quite liberating. Suddenly, lots of application processing can be done asynchronously without worrying about the overhead of maintaining the appropriate crontabs by hand.

Using a Phing filter to flush browser caches

nospam@example.com (David Winterbottom) — Sun, 15 Mar 2009 19:01:13 +0000

A quick Phing tip that's made my life easier when deploying new versions of commandlinefu.com.

One of the key performance recommendations from Steve Souders' excellent "High Performance Websites" is to use Expires HTTP headers to set far-future expiration dates for your site components (such as images, Javascript files and CSS stylesheets). This way, browsers can cache the files between requests giving a performance boost to your site. Assuming you're using Apache for serving, the following settings can be used to set these headers for all Javascript and CSS files (there are a few alternative ways of achieving the same result):

<FilesMatch "\.(ico|pdf|flv|jpg|jpeg|png|gif|js|css|swf)$">
    Header set Expires "Thu, 15 Apr 2010 20:00:00 GMT"
</FilesMatch>

The main issue to be aware of using this technique is that, when your components change, you need to ensure your visitors are forced to download the latest version rather than using the one cached by their browser. The only way to ensure this happens is to use a different URL for the assets in question. One option might be to rename the files themselves but a more convenient alternative is to include a query string as part of the request URL (eg <script src="/js/behaviour.js?2009-03-15" type="text/javascript" />). Then changing the query string component is sufficient to force browsers to make a full request for the new component.

This works well but is an easy-to-forget overhead for deployment. However, this substitution can be automated by making use of the Filters that Phing provides. Doing so is trivial: simply insert a tokenised string as the query string after your asset URLs. That is:

<link rel="stylesheet" href="http://codeinthehole.com/css/styles.css?~~CACHEBUSTER~~" type="text/css" />

where the '~~' delimiter indicates the token. Then include something like the following snippet in your Phing deployment script.

<tstamp>
    <format property="build.datetimestring" pattern="%Y-%m-%d-%H-%M" />
</tstamp>
...
<target name="create-temp-build" description="Creates a temporary copy of the source files">	
    <echo msg="Copying deployment files into temporary directory" />
    <copy todir="${dev.folder.temp}">
        <filterchain>
            <replacetokens begintoken="~~" endtoken="~~">
                <token key="CACHEBUSTER" value="${build.datetimestring}" />
            </replacetokens>
        </filterchain>
	<fileset refid="deployment-files" />
    </copy>
</target>

The filterchain component of the copy task parses the given fileset for matching tokens that match and replaces them with the given value. In this example, I'm using timestamps as the replacements as these will ensure a different query string on each deployment. Doing so ensures that the deployed HTML includes the lines:

<link rel="stylesheet" href="http://codeinthehole.com/css/styles.css?2008-03-15-21-51" type="text/css" />
<script type="text/javascript" src="http://codeinthehole.com/js/site-behaviour.js?2008-03-15-21-51">

which then ensure that all subsequent visitors download the latest versions of the CSS and javascript files. The above target is taken from a deployment script I use which creates a temporary snapshot of the codebase that I want to deploy, but the basic principle of using the replacetokens filter is easily transferable to any deployment script.

Deploying to a shared hosting environment using Phing

nospam@example.com (David Winterbottom) — Sun, 04 Jan 2009 20:18:50 +0000

Deploying applications to shared hosting environments can be a pain when no SSH access is provided. Consequently, it's hard to avoid using FTP to deploy files from your development environment to a production server. In such trying circumstances, it's easy to form self-destructive habits like using drag-and-drop FTP deployment - a very bad thing. Much better is to write an automated deployment script so that you can build to production in one clean step, a key tenet of The Joel Test for writing better code (highly recommended). Get deployment right at the start and making major changes to your application becomes trivial rather than a complete nightmare.

Phing is great for deploying PHP applications. It's well documented, easy to use and extend, and works well with other libraries such as PHPunit. Although it's not inthe current docs for version 2.3.x there is a useful FTP task (FtpDeployTask.php) in Phing 2.3.3 that can be used to create a simple build script for deploying to shared hosting environments using FTP. Syntax and usage are trivial given a glance through the source code (/usr/share/php/phing/tasks/ext/FtpDeployTask.php on Ubuntu systems).

Here's a simple build.xml deployment script for a CodeIgniter project that uses the FtpDeploy task:

<?xml version="1.0" ?>
<project name="Shared hosting deployment" default="deploy-application-files" basedir=".">

<property name="ftp.host" value="ftp.example.com" />
<property name="ftp.port" value="21" />
<property name="ftp.username" value="user" />
<property name="ftp.password" value="password" />
<property name="ftp.dir" value="/public_html/" />
<property name="ftp.mode" value="ascii" />


<fileset dir="." id="files.images">
<include name="images/**/*" />
<include name="favicon.ico" />
</fileset>
<fileset dir="." id="files.application">
<include name="system/application/**/*" />
<include name="css/*" />
<include name="js/*" />
</fileset>
<fileset dir="." id="files.system">
<include name="system/**/*" />
<exclude name="system/application/**/*" />
<include name="index.php" />
<include name="robots.txt" />
<include name=".htaccess" />
</fileset>


<target name="deploy">
<echo message="Copying fileset '${deploy.fileset.refid}' to ${ftp.host} in ${ftp.mode} mode" />
<ftpdeploy
host="${ftp.host}"
port="${ftp.port}"
username="${ftp.username}"
password="${ftp.password}"
dir="${ftp.dir}"
mode="${ftp.mode}">
<fileset refid="${deploy.fileset.refid}" />
</ftpdeploy>
</target>
<target name="deploy-images">
<echo msg="Deploying image files" />
<phingcall target="deploy">
<property name="deploy.fileset.refid" value="files.images" />
<property name="ftp.mode" value="binary" override="true" />
</phingcall>
</target>
<target name="deploy-application-files">
<echo msg="Deploying application files" />
<phingcall target="deploy">
<property name="deploy.fileset.refid" value="files.application" />
</phingcall>
</target>
<target name="deploy-system-files">
<echo msg="Deploying system files" />
<phingcall target="deploy">
<property name="deploy.fileset.refid" value="files.system" />
</phingcall>
</target>
<target name="deploy-all">
<phingcall target="deploy-images" />
<phingcall target="deploy-application-files" />
<phingcall target="deploy-system-files" />
</target>
</project>

As we're forced into using clumsy, out-dated FTP (rather than the wonderful rsync) to copy the files onto the production server, each deployment overwrites existing files with the selected fileset and this can be quite time-consuming for large filesets as it often moves across a load of files that haven't changed. Using FTP, this is tricky to avoid and probably not worth the effort. Instead, there are four separate deployment targets that only move specific subsets of the total application fileset over to the production environment (split according to the standard CodeIgniter directory structure) - most of the time we only want to move the "files.application" fileset across. These targets make use of the extremely useful "phingcall" target which allows a parameterised "deploy" target to be created which takes a fileset id as an argument.

With the above build.xml in the root of your project, deploying to production is now as simple as:

> cd /path/to/application/ && phing deploy-all

although most of the time the following suffices:

> cd /path/to/application/ && phing

Note that FtpDeploy.php has a dependency on the PEAR package Net_Ftp which can be trivially installed in the usual manner:

> sudo pear install Net_FTP

Of course, this script is overly simplistic and only works well with a single developer. However, it would be easy to extend to team development by integrating with SVN so that the latest revision (or designated tag/branch) is checked out into a temporary directory before being copied over to the production server.

Preferably however, get shared hosting that provides SSH access (nearlyfreespeech.net is great). Then you can deploy in a more sophisticated manner using SVN exports or rsync.