Code in the hole (Entries tagged as bash)

The most important command-line tip - incremental history searching with .inputrc

nospam@example.com (David Winterbottom) — Tue, 03 Feb 2009 23:40:00 +0000

Getting www.commandlinefu.com off the ground has renewed my interest in Bash, UNIX and all things command-line. Powerful one-liners are things of beauty and are worth collecting; however what I consider to be the most influential command-line tip I know covers four:

"\e[A": history-search-backward
"\e[B": history-search-forward
"\e[C": forward-char
"\e[D": backward-char

These lines need to be placed in your ~/.inputrc file, the start-up script for the Readline utility used by Bash (as well as several other applications) and others). The important commands here are the first two, which bind your up and down cursor keys to incrementally search your history. (The second two ensure that left and right continue to work correctly).

This is incredibly useful for retrieving commands you've used previously and makes a huge difference to your productivity. For instance, to find a previous SSH command from a few days ago, simply type "ss" and press up a few times. This will allow you to browse through all your previous ssh… commands until you find the right one - you never need to use more than 4 or 5 keystrokes to retrieve any previous command. If your cycling through too many commands to find the right one, type in a few more characters to refine the search.

As indicated above, this functionality is available in all applications that use Readline including MySQL, Python, IRB (interactive Ruby shell) and others. Once you're used to this feature, it's hard to live without - the first thing I do once I've been set up as a user on a new server is update my .inputrc file to contain these settings. The one place where I sorely wish this functionality existed is the Firebug Javascript commandline in Firefox (a ticket already exists requesting a similar feature).

Another way of searching your history is to use CTRL+R, which essentially performs a full-text search on your history (keep pressing CTRL+R to cycle through results). In this case, searching for "ssh" will locate all commands that feature this string anywhere in the command. Although this is actually a more powerful feature than the incremental history search described above, I don't often use it as: (a) the incremental search generally lets me jump to the desired command in fewer key-presses and (b) I find "ssh", CTRL+R, CTRL+R slightly awkward to type and less intuitive than "ssh", UP, UP. Horses for courses really - you could probably be just as efficient with either one.

» Hat-tip to the place where I first learnt this: Power Shell Usage by Simon Myers

One extra thing: this functionality can be neatly complimented by some choice history settings in your ~/.bashrc file:

export HISTSIZE=1000000
export HISTFILESIZE=1000000000

These simply set your history to be very large so that you have a huge bank of commands to search.

Monitoring MySQL with Ganglia and gmetric

nospam@example.com (David Winterbottom) — Sun, 14 Dec 2008 21:34:00 +0000

Following some server monitoring advice from the excellent 'Building Scalable Web Sites' by Cal Henderson, I've recently been experimenting with Ganglia on a cluster of servers at Tangent Labs. It has proved to be deeply impressive and has given us a great tool for keeping an eye on how our servers are performing, as well as providing an invaluable diagnostic tool should things go wrong.

In essence, Ganglia is a distributed monitoring application that allows statistics on a cluster of servers to be aggregated in a single place. Moreover, an excellent web front-end is also provided to view charts of the statistics being collected. You can view a live demo of the Ganglia monitoring application for the UC Berkeley Millennium Project which shows exactly what Ganglia can provide.

Out of the box, Ganglia collects basic server stats on load, memory, CPU, network and disk. However, what makes it great is that it can be easily extended to collect statistics on anything that you can measure from the command line. There is a repository of user-contributed scripts although it's not that extensive. In this post, I detail a script that can used to feed statistics about MySQL into Ganglia for monitoring - something important to any LAMP-stack application.

A few technical details first: Ganglia utilises three utilities to run each with a distinct responsibility:

gmond: A daemon responsible for collecting monitoring stats and sharing them using XML over TCP. The gmond daemon runs on all nodes that you want to monitor.
gmetad: A daemon responsible for collecting and storing data from other gmetad and gmomd services. This only needs to be running on a single, central node.
gmetric: A command-line application used for using Ganglia to monitor custom metrics.

All very simple really - see the docs for the nitty gritty. Ok, enough intro - here's my script for MySQL:

» gmetric-mysql.sh (3.1kb)

It's written in Bash and uses a temporary file to read the MySQL stats into before parsing and submitting to gmetric. This is done as many of the MySQL stats of interest are cumulative counts rather than rates (eg SELECTs per second) which are generally of more interest. Hence, the script estimates the rate of change by dividing the value delta by the timestamp delta.

To use this script, simply copy it into a convenient location, ensure it is executable and add the following line to your crontab.

* * * * * /path/to/gmetric-mysql.sh > /dev/null 2>&1

As soon as this is done, your Ganglia web front-end should start plotting the new metrics.

Checking all MySQL tables

nospam@example.com (David Winterbottom) — Sun, 23 Nov 2008 11:30:40 +0000

It's well known that MyISAM tables are prone to corruption and need to be regularly checked and repaired. Moreover, in a production environment, it can be beneficial to run a daily check of all tables and mail news of any errors to an appropriate developer/DBA.

There are two options for checking MySQL tables. The most effective method is to run the myisamchk utility directly on the index files (.MYI) of the tables in question (some simple shell expansion makes this easy):

myisamchk --silent --fast /path/to/datadir/*/*.MYI

However, this proses a problem in that you must ensure that no other programs are accessing the tables while they are being checked. Hence they must be locked, or better still, the MySQL daemon stopped before running any checks. Perversely, if this is not done, the act of checking the tables can corrupt them.

Another option is to use the CHECK TABLE syntax in SQL (which does not pose a risk of corruption). There are various scripts (written in PHP and bash) posted on the CHECK TABLE manual page but this operation can be done easily through a single line:

mysql -p<password> -D<database> -B -e "SHOW TABLES" \
| awk '{print "CHECK TABLE "$1";"}' \
| mysql -p<password> -D<database>

This dynamically creates a list of "CHECK TABLE ..." commands which is piped into MySQL for execution.

For checking a selection of tables rather than all, use the LIKE operator when selecting the tables to check:

mysql -p<password> -D<database> -B -e "SHOW TABLES LIKE 'User%'" \
| awk 'NR != 1 {print "CHECK TABLE "$1";"}' \
| mysql -p<password> -D<database>

This only checks the tables that start 'User'. Note that the awk program has an extra clause to ensure that the first line of MySQL output is skipped.

Monitoring MySQL processes

nospam@example.com (David Winterbottom) — Sun, 26 Oct 2008 22:06:00 +0000

Just a quick tip on monitoring the queries that mysql is handling on a production site. You can use the mysqladmin tool to return a list of the processes currently being handled. Combining this with the UNIX watch command allows a real-time monitoring of what's going on.

watch -n 1 mysqladmin processlist

The "-n 1" specifies that mysqladmin executes every second. Depending on your set-up, you may need to specify a mysql user and password:

watch -n 1 mysqladmin --user=<user> --password=<password> processlist

Following log files with tail -f

nospam@example.com (David Winterbottom) — Wed, 22 Oct 2008 09:03:00 +0000

UNIX is a majestic onion of discovery. Every day a new layer of understanding can be peeled away to give some new pungent goodness. Today's was the 'follow' option of the tail command.

It's commonplace to use tail for viewing the recent entries to a log file:

tail /var/log/apache2/error.log

Much more useful is set the 'follow' option so that, rather than echoing to STDOUT and returning control to the prompt - tail continues to watch the file in question and echos additional lines to the terminal. This can be very useful during development - I often leave a terminal open watching the error logs while I develop - so that I can quickly pounce on any errors that pop up.

tail -f my-app-error.log

Return control to the command prompt using CTRL+C (or run as a background process using &). This can be quite useful for monitoring the error logs of production sites where echoing error messages and warnings to the screen would be unacceptable.