Syslog-NG Log Processing on ATGRID

The syslog-ng software can generate a significant amount of data over time. On atgrid.grid.umich.edu we have a RAID5 system being used to store the logs in a MySQL DB (/data06/var/lib/mysql).

The syslog-ng system has a PHP script which does the equivalent of logrotate for the MySQL DB tables. The resulting DB tables are stored as logsYYYYMMDD.[frm|MYI|MYD].

I wrote a simple Perl script which uses the Date::Manip Perl package to migrate all such log tables older than 90 days off to /afs/atlas.umich.edu/accounting/atgrid. It is located with the other PHP scripts /var/www/html/php-syslog-ng/scripts/move_logs.pl. The file is:

#!/usr/bin/perl
#
#  Move tables older than 3 months to AFS
#
# Shawn McKee
#############################################

use Date::Manip;
$err=0;
$destdir="/afs/atlas.umich.edu/accounting/atgrid";
$srcdir="/data06/var/lib/mysql/syslog";
$maxdays=90;

@files = glob("$srcdir/logs*.MYD");
foreach $file (@files) {
#    print " Found file $file\n";
    $file=~/logs(\d\d\d\d)(\d\d)(\d\d)/;
    $year=$1;
    $mon=$2;
    $day=$3;
    $dateold = &ParseDate("$mon/$day/$year");
    $datenow = &ParseDate("today");
    $delta = DateCalc($dateold,$datenow,\$err);
    $dec=1;
    $ageindays = Delta_Format($delta,$dec,"%dt");
    if ($ageindays > $maxdays ) {
        $result=system("tar -jcf $destdir/logs$year$mon$day\.tar\.bz2 $srcdir/*$year$mon$day*");
#       print " Result of tar'ing ./*$year$mon$day* is $result\n";
        if ($result == 0) {
            $resdel=system("rm -f $srcdir/*$year$mon$day*");
#           print " Result of delete of $file is $resdel\n";
        }
        exit;
    }
}

This should be added to the cron.weekly run on atgrid.

-- ShawnMcKee - 01 Oct 2007