dCache Config Overview

dCache from Database to Filesystem

Below is a view of the chain of configuration for the dCache system starting with the PostgreSQL databases that back PNFS and going to the OS native filesystems that actually hold datafiles in the pools.

Database

PNFS uses a database to store file metadata. In our installation PostgreSQL is used.

PNFS provides the tool mdb for managing the databases.

List the databases used by PNFS, admin and data1 are setup by the installer:

root@msu2 ~# export PATH=/opt/pnfs/tools:$PATH
root@msu2 ~# mdb status
   ID   Name         Type    Status       Path  
 ---------------------------------------------- 
   0    admin         r     enabled (r)   /opt/pnfsdb/pnfs/databases/admin
   1    data1         r     enabled (r)   /opt/pnfsdb/pnfs/databases/data1

root@msu2 ~# ls -l /opt/pnfsdb/pnfs/databases/
total 0
-rw-r--r--  1 root root 0 Mar 27 00:14 admin
-rw-r--r--  1 root root 0 Mar 27 00:14 data1

Add a new one:

root@msu2 ~# mdb create test /opt/pnfsdb/pnfs/databases/test
root@msu2 ~# mdb update

Note The file /opt/pnfsdb/pnfs/databases/test is just a placeholder when PostgreSQL db is used. Still, it should be created (why?) if it isn't, use touch to make it.

You can see the database directly in PostgreSQL as well:

Welcome to psql 8.1.4, the PostgreSQL interactive terminal.

Type:  \copyright for distribution terms
       \h for help with SQL commands
       \? for help with psql commands
       \g or terminate with semicolon to execute query
       \q to quit

postgres=# \l
        List of databases
   Name    |   Owner   | Encoding 
-----------+-----------+----------
 admin     | postgres  | LATIN9
 billing   | srmdcache | LATIN9
 companion | srmdcache | LATIN9
 data1     | postgres  | LATIN9
 dcache    | srmdcache | LATIN9
 postgres  | postgres  | LATIN9
 replicas  | srmdcache | LATIN9
 template0 | postgres  | LATIN9
 template1 | postgres  | LATIN9
(9 rows)

postgres=# \c test
You are now connected to database "test".
test=# \d
        List of relations
 Schema | Name | Type  |  Owner   
--------+------+-------+----------
 public | pnfs | table | postgres

test=#\q

PNFS Directories

A directory in PNFS is linked to the database by issuing a PNFS "dot command". The command syntax is, .(Database ID)(Dirname), where Database ID is index from the mdb show command and the Dirname is the new directory to create. Example:

root@msu2 ~# cd /pnfs/msu-t3.aglt2.org
root@msu2 /pnfs/msu-t3.aglt2.org# mkdir '.(2)(testy)'

Directory tags

Once the directory is made, we need to add some "tags" to it that tell dCache how to use it.

The wormhole is a special tag used to tell dCache were to find config information for the directory in PNFS. Since we have one config area, we just want to copy the wormhole info from the main directory (which is established at install-time) into the new directory:

root@msu2 /pnfs/msu-t3.aglt2.org# source /usr/etc/pnfsSetup
root@msu2 /pnfs/msu-t3.aglt2.org# export PATH=/opt/pnfs/tools:$PATH
root@msu2 /pnfs/msu-t3.aglt2.org# echo ${shmkey}
1122

root@msu2 /pnfs/msu-t3.aglt2.org# sclient getroot ${shmkey} 0
0 000000000000000000001000 0000000000000000000010E0
root@msu2 /pnfs/msu-t3.aglt2.org# wormhole=`sclient getroot ${shmkey} 0 | cut -d' ' -f3`
root@msu2 /pnfs/msu-t3.aglt2.org# echo $wormhole
0000000000000000000010E0

root@msu2 /pnfs/msu-t3.aglt2.org# mdb show
   ID   Name         Type    Status       Path  
 ---------------------------------------------- 
   0    admin         r     enabled (r)   /opt/pnfsdb/pnfs/databases/admin
   1    data1         r     enabled (r)   /opt/pnfsdb/pnfs/databases/data1
   2    test          r     enabled (r)   /opt/pnfsdb/pnfs/databases/test

root@msu2 /pnfs/msu-t3.aglt2.org# sclient getroot ${shmkey} 2 ${wormhole}
2 000200000000000000001000 0000000000000000000010E0

Regular Tags

You can list them like this:

cat '.(tags)()'

grep "" `cat ".(tags)()"`

The storeName and sGroup tags are used to form the storage class string which has syntax storeName:sGroup@osm. The @osm part is the default for disk storage (vs. tape). The poolmanager configuration will reference these storage classes. These tags are established using cat or echo to send the info to a special "dot file" in PNFS.

echo "StoreName teststore" > '.(tag)(OSMTemplate)'
echo "testgroup" > '.(tag)(sGroup)'

Pool Groups

Pool groups just aggregate pools into larger units. Pools may be members of multiple pool groups. Pool groups and not pools are referred to in the storage class, so you have to have pool groups.

poolmanager config This is done in /opt/d-cache/config/PoolManager.conf. The poolmanager needs to be restarted to pickup changes to this file, do /opt/d-cache/bin/dcache-core restart (in there an alternate procedure?).

psu create pgroup <pool group name>
psu addto pgroup <pool group name> <pool name>

e.g.

psu create pgroup testpg
psu addto pgroup testpg testp

Pools

Single pools correspond one-to-one (AFAIK) to disks.

poolmanager config Note though that the pools need to be defined above where they get added to the pgroups in the config file...

psu create pool <a pool name>

e.g.

psu create pool testp

OS Native Filesystem

dCache pools should have their own filesystems, while this is not a requirement of dCache, we don't want another application impacting the pool. Wish to use a standard location for mounting these. Propose /dpool with /dpool/1, /dpool/2, etc for individual pools. This enumeration matches how the vdt installer does things. The dCache cells are running as root on the pool node, so permissions 700 should be sufficient on the pool, though 755 is fine.

Config files

site-info.def

DCACHE_POOLS="\
msu4.aglt2.org:all:/dpool/1 \
"

The installer will then create these files on the pool node:

/opt/d-cache/config/msu2.poollist

msu4_1  /dpool/pool1/pool  sticky=allowed recover-space recover-control recover-anyway lfs=precious tag.hostname=msu4

/opt/d-cache/etc/pool_path

/dpool/pool1 4188 no

You can move these mount points around on the pool node, keeping the pool name the same, msu_1 above (just need to stop and restart dCache pool and core services on the pool node).

Links

Links define dataflows in-to out-of and inside the the dCache system. They are defined in the poolmanager configuration.

Links are defined as the AND of conditions. Conditions are the OR of elementary conditions. In the poolmanager config file, elementary conditions are called "unit" and conditions are "ugroup".

The elementary conditions come in three types:

type	description
-net	IP/netmask of requesting host
-storage	storage class (of PNFS directory)
-dcache	like -storage, used for complex sites with tertiary storage

Link Groups

What are they?

Link Poolmanager Config

psu create unit <type> <elementary condition>
psu create ugroup <ugroup name>
psu addto ugroup <ugroup name> <elementary condition>

The above may seem redundant with the repetition on the , but note that the create unit command is needed to specify the condition type.

psu create link <link name> <ugroup name>
psu set link <link name> -readpref=<int> -writepref=<int> -cachepref=<int>
psu add link <link name> <pool group name>

Note Every link has to have some condition. You can use a -net 0.0.0.0/0.0.0.0 as a "condition" that is always true.

Read/Write For a system without tertiary storage, readpref and cachepref values will be equal.

Replication

-- TomRockwell - 07 Apr 2008