Using CVMFS at AGLT2

General Information

CVMFS is a new method of distributing ATLAS software that relies on using central repositories of software on servers located at CERN, BNL and other locations. This means that separate installations of software need not be made at each and every site where it is desired. Instead, the central repository is updated, and a simple kit validation job is sent to remote locations.

Access to the software makes use of a "squid" at the remote site (eg, at AGLT2). This squid pulls files from the central servers and caches them in its store as file requests comes to it. It, in turn, sends the files to the local, requesting worker node, where they are cached in a 25GB store on the worker. The first such request from the central servers may be a bit slow, depending on load, but each additional request is local to the AGLT2 network, and proceeds quickly out to the individual worker node. Future request for the same files from that worker are even faster as the data is now local to the worker, and the network transfer is now bypassed.

This is in contrast to the mechanism we have used until now at AGLT2, where the releases are stored on an NFS server, and network traffic is always involved in accessing them.

The descriptions below are valid for any locally submitted job, using CVMFS, and running on our cluster. They should also work for grid submissions using, eg, prun. Analysis jobs submitted through pathena will not use these setups, but should instead proceed transparently to work from the users point of view, once we have made the full transition to CVMFS.

NOTE: We can find no way to make the traditional/NFS releases work at the same time as the CVMFS releases. The plan is that those traditional releases will be deleted from our systems. However, this deletion will not come immediately, and we will entertain requests to retain them in some form if we can find a way to do so.

What versions of software are available in CVMFS?

After the setup file is sourced, then you can use 'showVersions' to see the list of installed software available.

• source /usr/local/bin/setup/cvmfs_atlas.sh
• showVersions

In general, Panda also keeps a knowledge of available releases at all sites here

Directions from Shawn

For local jobs, the following steps should be followed

• source /usr/local/bin/setup/cvmfs_atlas.sh
• Make sure you have a ".asetup" file in your home directory. It should have been created by the first step.
  • Edit the ".asetup" file, changing the "testarea" to some working directory you use for ATLAS
• Set up athena
  • asetup 16.6.5 (or whatever release you want to set up)
• There are other options you can set to provide access to tools/software.
  • Run "atlasLocalRootBaseSetup" to see the list
  • This is the same list that is printed when cvmfs_atlas.sh is sourced

Directions from TieSheng

• source /usr/local/bin/setup/cvmfs_atlas.sh
• Make sure you have a ".asetup" file in your home directory. It should have been created by the first step.
  • Edit the ".asetup" file, changing the "testarea" to some working directory you use for ATLAS
  • Change both projtest and multi settings to "False" from the default "True"
• Make yourself a setup.sh, containing something like the following content
  • Source this file before running Athena

setup_CURRDIR=`/bin/pwd`
setup_BASEDIR=/atlas/data18c/daits/p16.6.7.3
DBVERSION=""

source /usr/local/bin/setup/cvmfs_atlas.sh
cd ${setup_BASEDIR}
unset CMTPATH
if [ "${DBVERSION}" = "" ]; then
  asetup --standalone --usedcache --cmtconfig=i686-slc5-gcc43-opt --testarea=${setup_BASEDIR} 16.6.7.3,AtlasProduction
else
  asetup --standalone --usedcache --cmtconfig=i686-slc5-gcc43-opt --testarea=${setup_BASEDIR} --dbrelease=${DBVERSION} 16.6.7.3,AtlasProduction
fi
echo ${CMTPATH}
cd ${setup_CURRDIR}

Directions from Jonathan Long

For some reason, cvmfs_atlas.sh doesn't set the environment variables on condor, and so things like asetup and localSetupROOT don't work (but somehow ATLAS_LOCAL_ROOT_BASE does, I don't know why), I got around this by copying the alias from a local setup.

Using Athena

#   The .asetup file is in $HOME, and this is not otherwise found in a Condor job
export HOME=/afs/atlas.umich.edu/home/longjon
source /usr/local/bin/setup/cvmfs_atlas.sh
#  Taken from the interactive alias    "asetup"
source $AtlasSetup/scripts/asetup.sh --tag=16.0.3 

Using root from cvmfs

#   The .asetup file is in $HOME, and this is not otherwise found in a Condor job
export HOME=/afs/atlas.umich.edu/home/longjon
source /usr/local/bin/setup/cvmfs_atlas.sh
#   Taken from the interactive alias    "localSetupGcc"
source ${ATLAS_LOCAL_ROOT_BASE}/packageSetups/atlasLocalGccSetup.sh --gccVersion gcc435_x86_64_slc5
cd /cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/root/5.30.00-slc5-gcc4.3-i686/; source bin/thisroot.sh; cd -

Python

Something interesting to try:

• localSetupPython --pythonVersion=2.6.5-x86_64-slc5-gcc43

Example: Running the cvmfs_atlas.sh script

[ball@umt3int01:~]$ source /usr/local/bin/setup/cvmfs_atlas.sh
...Type localSetupDQ2Client to use DQ2 Client
...Type localSetupGanga to use Ganga
...Type localSetupGcc to use alternate gcc
...Type localSetupGLite to use GLite
...Type localSetupPacman to use Pacman
...Type localSetupPandaClient to use Panda Client
...Type localSetupROOT to setup (standalone) ROOT
...Type showVersions to show versions of installed software
...Type asetup [--help] to setup a release
...Type changeASetup [--help] to change asetup configuration
...Type diagnostics for diagnostic tools
...Type helpMe for help

References

• The User Interface ATLASLocalRootBase
• The AtlasSetup environment setup package
• asetup and what it is / does
• asetup reference
• Help (eGroup - see forum description in links for usage):
  • release kit problem
  • offline software help

-- BobBall - 31 Aug 2011