Reworking AGLT2's Logging Setup

In upgrading atgrid we have an opportunity to migrate from syslog-ng and php-syslog-ng to something new. The ELK stack (Elasticsearch, Logstash and Kibana) seems to be very powerful and gaining widespread adoption. In addition rsyslog seems to be a good choice for the basic logging on our nodes.

This page will document the steps and details in migrating AGLT2 over to this new software.

Upgrading ATGRID

The atgrid.grid.umich.edu host is currently an SL5 VM in VMware with 12 GB of ram, two virtual disks (250 and 160GB), and 4 processors. To "upgrade" I created a new VM with 12GB of ram, one 400GB virtual disk and 4 processors. I then configured Cobbler to provision this new VM using a server-large template. By PXE booting the VM Cobbler was able to setup and install SL6.5 64-bit on the atgrid-new VM.

After provisioning I need to take care of a few things. First I installed the current VMware tools for vSphere 5.5:

mount /dev/cdrom /mnt/cdrom/
cp /mnt/cdrom/VMwareTools-9.4.0-1399439.tar.gz ./
umount /mnt/cdrom
tar -zxvf VMwareTools-9.4.0-1399439.tar.gz
cd vmware-tools-distrib/
./vmware-install.pl

Then I added the repositories for rsyslog, elasticsearch and logstash

• rpm --import http://packages.elasticsearch.org/GPG-KEY-elasticsearch
• wget http://rpms.adiscon.com/v7-stable/rsyslog.repo
• Then setup two repos

[elasticsearch-1.0]
name=Elasticsearch repository for 1.0.x packages
baseurl=http://packages.elasticsearch.org/elasticsearch/1.0/centos
gpgcheck=1
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch



[logstash-1.3]
name=logstash repository for 1.3.x packages
baseurl=http://packages.elasticsearch.org/logstash/1.3/centos
gpgcheck=1
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch

Then I needed to install rsyslog and remove syslog-ng

• yum install rsyslog
• yum remove syslog-ng

One issue is the server-large template puts most of the space in '/' but the logging software typically uses /var to host logs. I created a new /rsyslog area and edited /etc/rsyslog.conf to include

$WorkDirectory /rsyslog/var # where to place spool files

and

# a new file will ben created daily because of the date in the filename.
$template DailyPerHostLogs,"/rsyslog/logs/%HOSTNAME%/%HOSTNAME%.%$YEAR%-%$MONTH%-%$DAY%.log"
*.* -?DailyPerHostLogs;TraditionalFormat

Next I created a new logrotate entry to handle compressing the rsyslog host logs after they are 7 days old.

• emacs -nw /etc/cron.hourly/compress-rsyslog.cron

# Compressing remote syslog messagefiles, older then one week using bzip2

find /rsyslog/logs -type f -mtime 7 -name "*.log" -exec bzip2 '{}' \;

At this point I rebooted and let CFengine3 complete the update of atgrid-new.

After reboot I noticed CFEngine had (properly) removed rsyslog and reinstalled syslog-ng. I turned off cfengine3 until I could update the configuration for atgrid and resinstalled rsyslog.

Additional installs:

• yum install rsyslog-elasticsearch.x86_64 rsyslog-mmjsonparse.x86_64 rsyslog-mmnormalize.x86_64 rsyslog-relp.x86_64 rsyslog-snmp.x86_64 rsyslog-gssapi.x86_64
• yum install elasticsearch
• /sbin/chkconfig --add elasticsearch
• wget wget https://download.elasticsearch.org/kibana/kibana/kibana-latest.tar.gz
• tar xzf kibana-latest.tar.gz
• mv kibana-latest /var/www/
• restorecon -R /var/www/kibana-latest
• yum install java-1.7.0-openjdk tar wget vim policycoreutils-python zip

I had to make sure that port 80 and 443 were opened in iptables. I then had to edit the elasticsearch.yml to define the "Cluster" and "Node" names

• emacs -nw /etc/elasticsearch/elasticsearch.yml

Then install some plugins

• /usr/share/elasticsearch/bin/plugin -install karmi/elasticsearch-paramedic
• /usr/share/elasticsearch/bin/plugin -install royrusso/elasticsearch-HQ
• //usr/share/elasticsearch/bin/plugin -install mobz/elasticsearch-head

Configure Logstash

• emacs -nw /etc/logstash/conf.d/rsysloginput.conf

RSYSLOG Details

We are migrating to using rsyslog on our systems instead of syslog-ng. To do this we need to determine how best to configure it.

-- ShawnMcKee - 06 Mar 2014