Track Transfers to dCache

Just some quick notes on tracking specific transfers to dCache.

• First, make sure the SRM logging is verbose enough (going to catalina.out)
  • Login to the dCache admin interface
  • 'cd RemoteHttpTransferManager' or 'cd SrmSpaceManager'
  • 'log4j appender set detailed INFO'

At this point there should be lots of details being sent to /opt/d-cache/libexec/apache-tomcat-5.5.20/logs/catalina.out.

You will need to know the file name of the file being transferred, e.g., /pnfs/aglt2.org/atlasmcdisk/myfile.out.

Just grep for this in the catalina.out to see the details about the transfer.

From the dCache admin interface you can cd to 'PnfsManager' and do:

• 'pnfsidof /pnfs/aglt2.org/atlasmcdisk/myfile.out' (or whatever name your file is)

This will get you the pnfsid of the destination file. Our dCache is setup to allow passive FTP transfers which means the destination pool participates in the transfer. The 'grep' of the catalina.out should should the GridFTP door that is initially contacted for the transfer, but to find out which node is actually doing the transfer you need to locate the pnfsid. There are two ways to do this:

• Try ssh commands (via key access) to the set of pool nodes doing

  ls /dcache*/pool/data/<pnfsid>

• Examine via lsof on each pool node:

  lsof | grep <pnfsid>

lsof | grep 000073E5A0045DD446BCB80A307AB8EB8A56
java       384612      root  153u      REG               8,32 2995682633  268444003 /dcache1/pool/data/000073E5A0045DD446BCB80A307AB8EB8A56

This will give you the PID of the process doing the transfer. You can search netstat to see what this process is doing:

[root@umfs12 ~]# netstat -apn | grep 384612
tcp        0      0 0.0.0.0:33134               0.0.0.0:*                   LISTEN      384612/java
tcp        0      0 0.0.0.0:22169               0.0.0.0:*                   LISTEN      384612/java
tcp        0      0 0.0.0.0:21820               0.0.0.0:*                   LISTEN      384612/java
tcp        0      0 192.41.230.32:22169         192.12.15.232:35971         ESTABLISHED 384612/java
tcp        0 401096 192.41.230.32:37958         192.41.236.51:42178         ESTABLISHED 384612/java
tcp        0      0 192.41.230.32:41579         192.41.230.44:40709         ESTABLISHED 384612/java
tcp        0      0 192.41.230.32:21820         192.12.15.235:53702         ESTABLISHED 384612/java

You could then use tcpdump to get the details of the ongoing transfer:

• 'tcpdump -i eth2 host 192.12.15.232 and src port 35971 and dst port 22169' (This picks a particular transfer)

You can save it to a file via '-w filename'.

-- ShawnMcKee - 12 Mar 2010