Setting up ATLAS Area HOTDISK for AGLT2

As of mid September 2009 we need to provide a new space-token area in Tiers of ATLAS call 'HOTDISK'. There are a few components to this:

• Identify the storage to be used
• Create the area in dCache/PNFS
• Assign a space token
• Create a new ToA entry

Our site uses dCache and these notes are specific to dCache implementations.

Identify the Storage to be Used

The 'HOTDISK' area is intended to be used for high-demand files. As such files in this area should be stored in such a way as to support a large number of clients consistently accessing them. In dCache this is best done by replicating the file into multiple pools. When a client needs these files it has many possible candidates and can benefit from having many servers able to provide the file.

At AGLT2 we have the most pools in our MCDISK area. This is setup as a specific link-group in dCache: atlasmcdisk-link-group (link ID 210000). We also have a service (created by Wenjing Wu) called *rephot* which scans for "hot" files and makes one or more replicas (depending upon demand). As files "cool" it removes replicas. We would like to leverage this capability for HOTDISK. Since rephot is running on all storage areas already, once HOTDISK exists it will run on it as well.

Thus, it would be best to just "reuse" this existing set of pools for HOTDISK if possible.

Create the Area in dCache/PNFS

We need to create a new top-level directory for HOTDISK in our existing PNFS area. We login to head02, mount pnfs and:

$ mkdir /pnfs/aglt2.org/atlashotdisk
$ cd /pnfs/aglt2.org/atlashotdisk

$ echo "atlas">".(tag)(sGroup)"
$ echo "StoreName usatlas" > ".(tag)(OSMTemplate)"
$ echo  "ONLINE" > ".(tag)(AccessLatency)"
$ echo  "REPLICA" > ".(tag)(RetentionPolicy)"

$ grep "" $(cat ".(tags)()")
.(tag)(OSMTemplate):StoreName usatlas
.(tag)(sGroup):atlas
.(tag)(AccessLatency):ONLINE
.(tag)(RetentionPolicy):REPLICA

This will connect the directory with the 'atlas' storage group which is what we used for MCDISK. At this point, any files going to /pnfs/aglt2.org/atlashotdisk will end up on the same pools as MCDISK.

NOTE TBD is what postgres DB this is using? We have multiple ones setup for each area.

Assign a Space Token

One problem we have had at AGLT2 is setting up space-tokens based upon VO and Role. For example the AGLT2_MCDISK area currently supports the following space-tokens:

Space-token Description	Space-token	VoRole	Reserved	Used	Available
ATLASMCDISK	570010	/atlas/usatlas/Role=production	50291	9016	41274
ATLASMCDISK	570000	/atlas/Role=production	141561	126872	14688

When we configured space reservations and then setup ATLAS space-tokens at AGLT2 we thought we needed to utilize "VOs" and "Roles" to provide the mapping to the space-token. However it seems like it may be possible to use a system 'user' instead of a VOGroup/VORole. This would be a great advantage in that multiple roles can map to a single user and this mapping is controlled elsewhere (via GUMS/VOMS). For example, as shown above, we need to provide access to both /atlas/Role=production and /atlas/usatlas/Role=production users. This would be already handled if we could instead use usatlas1 since both of those VOGroup/VORoles already map to usatlas1. We tried to get this to work but were unable to. See below.

To set this up for HOTDISK we tried the following:

reserve -vog=usatlas1 -vor=* -acclat=ONLINE -retpol=REPLICA -desc=ATLASHOTDISK -lgid=210000 5TB "-1"

However this fails:

[head01.aglt2.org] (SrmSpaceManager) admin > reserve -vog=usatlas1 -vor=*  -acclat=ONLINE -retpol=REPLICA -desc=ATLASHOTDISK -lgid=210000 5TB "-1"
Link Group 210000 Name:atlasmcdisk-link-group FreeSpace:41738185868023 ReservedSpace:-167496838425593 AvailableSpace:209235024293616 VOs:{/atlas:production}{/atlas:*}{/atlas/soft-valid:production}{/atlas/usatlas:*} onlineAllowed:true nearlineAllowed:true replicaAllowed:true custodialAllowed:false outputAllowed:true UpdateTime:Sat Sep 19 13:59:21 EDT 2009(1253383161688) is found, but it cannot accommodate the reservation requested,
check that the link group satisfies the following criteria:
         it can fit the size you are requesting (5000000000000)
         vogroup,vorole you specified (usatlas1,*) are allowed, and
         retention policy and access latency you specified (REPLICA,ONLINE) are allowed

Checking the linkgroup:

[head01.aglt2.org] (SrmSpaceManager) admin > ls -lgid=210000
Found LinkGroup:
210000 Name:atlasmcdisk-link-group FreeSpace:41738185868023 ReservedSpace:-167496838425593 AvailableSpace:209235024293616 VOs:{/atlas:production}{/atlas:*}{/atlas/soft-valid:production}{/atlas/usatlas:*} onlineAllowed:true nearlineAllowed:true replicaAllowed:true custodialAllowed:false outputAllowed:true UpdateTime:Sat Sep 19 13:59:21 EDT 2009(1253383161688)
570000 voGroup:/atlas/usatlas voRole:production retentionPolicy:REPLICA accessLatency:ONLINE linkGroupId:210000 size:10000000000000 created:Wed May 21 14:05:42 EDT 2008 lifetime:-1ms expiration:NEVER description:ATLASMCDISK state:RESERVED used:9681415959211 allocated:0
570010 voGroup:/atlas voRole:production retentionPolicy:REPLICA accessLatency:ONLINE linkGroupId:210000 size:140000000000000 created:Wed May 21 14:05:42 EDT 2008 lifetime:-1ms expiration:NEVER description:ATLASMCDISK state:RESERVED used:136228570084318 allocated:4088053286

The FreeSpace is over 41TB. I'm not sure why we are getting the "...cannot accomodate the reservation requested". OK found it. The reserve command is not allowing me to use -vog=<username>. For now I will setup for /atlas/Role=production:

reserve -vog=/atlas -vor=production -acclat=ONLINE -retpol=REPLICA -desc=ATLASHOTDISK -lgid=210000 5TB "-1"

Now the link-group reservations look like:

Space-token Description	Space-token	VoRole	Reserved	Used	Available
ATLASHOTDISK	10063094	/atlas/Role=production	5000	0	5000
ATLASMCDISK	570000	/atlas/Role=production	170000	136000	34000
ATLASMCDISK	570010	/atlas/usatlas/Role=production	10000	9016	984

Create a New TiersOfATLAS Entry

The existing entry for AGLT2_MCDISK looks like:

    'AGLT2_MCDISK':
    {
        'domain': '.*aglt2.org.*/atlasmcdisk/.*',
        'email': 'smckee@umich.edu',
        'toolAssigner': 'lcg',
        'fts': BNLFTS,
        'srm': 'token:ATLASMCDISK:srm://head01.aglt2.org:8443/srm/managerv2?SFN=/pnfs/aglt2.org/atlasmcdisk/',
        'alternateName' : ['AGLT2'],
    },

So we propose creating the new ToA for AGLT2_HOTDISK:

    'AGLT2_HOTDISK':
    {
        'domain': '.*aglt2.org.*/atlashotdisk/.*',
        'email': 'smckee@umich.edu',
        'toolAssigner': 'lcg',
        'fts': BNLFTS,
        'srm': 'token:ATLASHOTDISK:srm://head01.aglt2.org:8443/srm/managerv2?SFN=/pnfs/aglt2.org/atlashotdisk/',
        'alternateName' : ['AGLT2'],
    },

As long as the space-token description ATLASHOTDISK exists and is assigned correctly this should work.

-- ShawnMcKee - 18 Sep 2009