Monitor Disk Activity with iostat and Ganglia

The iostat utility from the sysstat package provides information about disk operations and throughput. It works in two ways, one is that it records system activity and keeps historical information that can be retrieved using the command line tool, the second is providing current statistics via a command line tool. For systems that are impacted by disk performance, this information is much more detailed that the existing ganglia metrics. Described here is a ganglia plugin that pushes the statistics from iostat into ganglia.

Sysstat package

The sysstat package is included in Scientific Linux, but may not be installed by default. Installing it enabled the recording of performance data for the node.

Ioststat

When the iostat program is run, the first set of output is historical (averaged since the node was booted), the following sets of data are the current ones. The interval that iostat averages over can be specified, the default (and shortest possible) period is one second. Note that we have to wait for the averaging interval amount of time to get the current stats. Here is an example run:

Ganglia Python Plugin

• http://sourceforge.net/apps/trac/ganglia/wiki/ganglia_gmond_python_modules
• http://sourceforge.net/apps/trac/ganglia/browser/trunk/monitor-core/gmond/modules/python/README.in README about Python plugins
• http://sourceforge.net/apps/trac/ganglia/browser/trunk/monitor-core/gmond/python_modules/network/tcpconn.py Python plugin that I referenced when writing this one
• http://ganglia.sourceforge.net/gmetric/ List of user contributed gmetrics on official site
• http://ben.hartshorne.net/ganglia/ some misc ganglia plugins on a random website
• http://monami.sourceforge.net/userguide/ch03s05.html Monami aims to be a sort of monitoring switch board, works with ganglia

Newer versions of ganglia (3.1+?) have support plugins. All of the usual metrics are now implemented with plugins. One of the C language plugins is a Pythia master plugin; users may write plugins in either C or Python.

To use a Python plugin, one writes a Python script that supports three specified routines (metric_init, ??) and a callback routine(s) that will provide the actual data to ganglia when needed. The metrics are then setup via a config file, this config file specifies the usual parameters for metrics as well as the name of the callback routine that will provide the data.

Iostat can provide 11 statistics on each disk device on the system --- probably we only want to push selected values to ganglia. This plugin provides flexibility in that which values are stored is configured in the config file --- the Python script doesn't need to be modified. The "name" parameter in the config file is used to do this. A single callback routine is used, it parses the name based on "device_stat" where device is like "sda", "sdb", "sdb1" etc. and stat is like "rhz", "whz". Device refers to the disk or partition and stat refers to one of the output from iostat.

To see the script and example config file, see our SVN repo:

(yet to enter it)

Details on the script

The interval that iostat averages over can be set. The script caches the output from iostat so that calls to the command are minimized. Note that each call to iostat needs interval time to complete. This doesn't wait doesn't seem to harm ganglia, the script could be modified to do this calling and caching in a separate thread to hide or eliminate this delay.

It would be nice to extend the script so that disks can be specified by mount point instead of device. Suggest simply dropping slashes from mount points, so the read rate of /dcache would be specified as "dcache_rhz". The init routine should figure out and remember the mount point to device mapping.

Bandwidth is reported in kilobytes per second (rkbhz and wkbhz).

The script should have some error checking added, such as is iostat command available...

Will deploy the plugin to some non-critical nodes first, don't think performance will be an issue, but wish to verify at some level.

The config needs to specify the stat data type in the initialization. For this plugin, all the stats are "float" type. I had a problem with not having the "format" specified as the matching "%f". This is more of a development issue, as it won't need to be modified by the user.

-- TomRockwell - 24 Mar 2010