This page is OBSOLETE All site-services now run at BNL.

AGLT2 DQ2/DDM Verification and Debugging

One of the central tasks for our Tier2 is to support a DQ2 server. The notes below are to help with verifying and debugging our DQ2 installation on the AGLT2.

There is also a nice page at LRCTesting you can refer to.

Important Configuration Information

Our DQ2 "server" has two components. The HTTP frontend is hosted on linat10.grid.umich.edu (SLC4, 32 bit), while the LFC (Logical File Catalog) are hosted on umfs02.grid.umich.edu (SLC4, 64 bit) on /atlas/data08/dq2. The two primary storage locations are umfs05.aglt2.org (SLC4, 64 bit) on /atlas/data16/dq2 (GridFTP) and head01.aglt2.org (SLC4, 64 bit) on /pnfs/aglt2.org/dq2 (SRM/dCache). Our DQ2 services run on dq2.aglt2.org (SLC4, 32 bit) while our site primary gatekeeper is gate01.aglt2.org (SLC4, 64 bit) running OSG V0.8.0 software.

• HTTP for DDM: linat10.grid.umich.edu
• LRC: umfs02.grid.umich.edu
• Gatekeeper: gate02.grid.umich.edu

The UMFS02 and LINAT10 machines both have a account 'dq2' setup (homespace in AFS). The installed software is at /opt/dq2 on both machines. To access the account login as root and 'su - dq2'. Then 'cd /opt/dq2' and 'source config/AGLT2/environment.sh'.

Monitoring information

The Tier2 services are monitored by Cacti at: Cacti monitor

The Distributed Data Management (DDM) monitor is at DDM for AGLT2

The PANDA monitoring for production is at PANDA Production Monitoring

Verification/Debugging

This section is based upon an email from Dan Schrager about verifying our DQ2 setup.

There are a number of items to check:

• From UMFS02, check the myproxy proxy on myproxy.usatlas.bnl.gov: myproxy-info -s myproxy.usatlas.bnl.gov -d
• Verify the gridftp service is accessible on UMFS02: telnet umfs02.grid.umich.edu 2811 (You should get a prompt)
• Verify the dq2 processes are running on linat01: ps -efww | grep dq2 (You should see mysqld, httpd and (many) python commands running)
• As the dq2 user do crontab -l and verify the crontab contents
• From UMFS02, check the mysql database:
  • mysql -udq2user -pdqpwd -h `hostname -f`
  • show databases;
  • user localreplicas;
  • show tables;
  • select * from t_lfn;
  • quit;
  • Example results here
• Query from a browser:
  • http://linat10.grid.umich.edu:8000/dq2/storages/default
  • http://linat10.grid.umich.edu:8000/dq2/monitor.py/isAlive
  • http://linat10.grid.umich.edu:8000/dq2/lrc/endpoint
  • http://linat10.grid.umich.edu:8000/dq2/lrc/PoolFileCatalog?lfns=lib.0109724d-0c7d-4f1a-ae0c-d3ebf3a2e85b.tgz

To check current status tail the relevant log files (on dq2.aglt2.org):

• tail -f /var/log/dq2/dq2.log

Init/Renew MyProxy Credentials

As the 'dq2' user setup your environment on dq2.aglt2.org

• cd /opt/dq2
• source setup.sh
• = source /opt/OSG080/setup.sh=

You can then initialize or renew your credentials:

Old way: (non space token enabled srm doesnot require the voms extention)

• grid-proxy-init -valid 720:0
• myproxy-init -t 720 -c 720 -s myproxy.usatlas.bnl.gov -d

New way: (space token enabled srm requires the voms extention)

• voms-proxy-init --voms atlas:/atlas/usatlas/Role=production

Be sure to use the correct MyProxy password for FTS. See config files

Check things:

• myproxy-info -s myproxy.usatlas.bnl.gov -d
• grid-proxy-info
• voms-proxy-info -all

Recreating the 'queued_transfers_AGLT2' database

To do this you need to stop all DQ2 site-services. First edit the existing crontab for 'dq2' user and comment out the various services.

Next stop all dq2 processes currently running.

Then login to the mysql db:

mysql -u<user> -p<password> -h umfs02.grid.umich.edu queued_transfers_AGLT2

See the executor_conf.py to get the correct user and password.

Then delete and recreate the DB:

mysql> drop database queued_transfers_AGLT2;
Query OK, 80 rows affected (4.64 sec)

mysql> create database queued_transfers_AGLT2;
Query OK, 1 row affected (0.00 sec)

mysql> show databases;
+------------------------+
| Database               |
+------------------------+
| information_schema     | 
| queued_transfers_AGLT2 | 
| test                   | 
+------------------------+
3 rows in set (0.00 sec)

The last step is to just run the RecreateDatabase.py script:

 python DQ2/siteservices/executor/classic/RecreateDatabase.py

NOTE: Don't forget to re-enable the crontab entry for user 'dq2'!

Other Important URLs

There are a few other URLs which provide information at:

• DDM Page at CERN
• DDM Developers Page at CERN, includes useful tutorial presentations. Also linked from above page.

Please send Shawn McKee any edits or additional URLs that are relevant.

-- ShawnMcKee - 13 Dec 2006