OSG CE 0.4.1 Install Instructions

Introduction

This document is intended for administrators responsible for installing and configuring:

• OSG Compute Element (CE) version 0.4.1 onto OSG Production Resources
  

It is not meant as an all-inclusive guide to Grid computing or even all the options for configuring a CE.

Conventions used

The following conventions are used in these pages:

• mono-spaced text indicates terminal output
• bold mono-spaced text indicates terminal input (by the user)
• mono-spaced text in italics indicates variable data that may differ on your installation

• the > prompt indicates commands that do not require a specific shell or root access
• the # prompt indicates commands that require root access
• the $ prompt indicates sh- or bash-specific commands

• the % prompt indicates csh- or tcsh-specific commands

Major updates:

Operating Systems

If you experience problems with the installation of VDT supported software or for general system requirements, the basic VDT 1.3.10b System requirements page may be useful.

Installation Method

The installation and configuration method is based on Pacman. Pacman is a package manager like RPM or dpkg, but is able to work on and support multiple platforms. Specific Pacman directions are given below, and in the pre-installation procedures.

Assistance during the installation

If you have questions during the installation of this software, the first line of support is your VO Support Center.

Alternative options for less formal queries and assistance are:

1. Knowledge Base
   
   The IU OSG Knowledge Base can be found here. Typing OSG as your search string allows you to browse all titles.

2. Mailing lists
   The mailing list osg-general@opensciencegrid.org is available for you to submit questions.
   The osg-general mail list archives are available for viewing.
   More information about the mailing lists is available at OSG Mailing Lists.

3. Community Chat
   Real-time help is also available in the osg-communitychat room .

What's getting installed?

OSG 0.4.1

VDT 1.3.10b

You have installed a subset of VDT version 1.3.10b:
    CA Certificates v13 (includes IGTF 1.1 CAs)
    EDG CRL Update 1.2.5
    EDG Make Gridmap 2.1.0
    Fault Tolerant Shell (ftsh) 2.0.12
    Generic Information Provider 1.0.15 (Iowa 15-Feb-2006)
    Globus Toolkit, pre web-services, client 4.0.1
    Globus Toolkit, pre web-services, server 4.0.1
    Globus Toolkit, web-services, client 4.0.1
    Globus Toolkit, web-services, server 4.0.1
    GLUE Schema 1.2 draft 7
    GPT 3.2
    Java SDK 1.4.2_10
    KX509 20031111
    Logrotate 3.7
    MonALISA 1.4.12
    MyProxy 3.4
    MySQL 4.1.11
    PPDG Cert Scripts 1.7
    PRIMA Authorization Module 0.3
    RLS, client 3.0.041021
    UberFTP 1.18
    Virtual Data System 1.4.4

/etc/xinetd.d services

gsiftp
globus-gatekeeper

/etc/init.d services

edg-crl-upgraded
gris
globus-ws
mysql         (used by globus-ws)
condor        (setup optionally by ManagedFork)
MLD           (setup by configure-osg.sh)

cron services

mis              (MIS-CI)
root             (addition of entry for vdt log rotation)

/etc/services

Additional entries:

gsiftp  2811/tcp        # Added by the VDT
globus-gatekeeper       2119/tcp        # Added by the VDT

Pre-installation Checklist

System or Cluster Already Setup including a functioning batch system

It is assumed that your hardware is already running one of the operating systems previously mentioned. If your system is a cluster, the batch queuing system should likewise previously be installed and configured. If you intend to use the Condor system, that can be installed with pacman from the VDT cache before this grid middleware. It is strongly recommended to install Condor in a different directory than your main CE installation.

To pull Condor from the VDT cache, use:

pacman -get http://vdt.cs.wisc.edu/vdt_1310_cache:Condor

The installation will ask for local storage space for 3 separate purposes:

• OSG applications
• data
• temporary caching.

For details on the storage use and configuration, please reference the section on Local Storage Configuration in this guide.

Preserving a pre-existing Condor installation?

If you have installed Condor as your batch queuing system, you are required to set two environment variables so the Condor path is correctly configured.

• VDTSETUP_CONDOR_LOCATION - the location of your Condor installation (e.g. /opt/condor). The Condor bin/, bin, etc/, lib/... directories should be directly under this location. (Our VDTSETUP_CONDOR_LOCATION = /opt/condor-6.7.19 )

• VDTSETUP_CONDOR_CONFIG (optional): The location of your Condor configuration file (if non-standard). Default is $VDTSETUP_CONDOR_LOCATION/etc/condor_config. (Our VDTSETUP_CONDOR_CONFIG=/opt/condor-6.7.19/etc/condor_config )

This will result in:

• VDT middleware services pointing to appropriate locations in your external Condor installation.

For example, if you wish to preserve a Condor installed in the standard location,

  $ export VDTSETUP_CONDOR_LOCATION=$CONDOR_ROOT

will suffice (provided $CONDOR_ROOT is set, per the usual Condor config).

Compilers

The VDT installation presumes that you have a working C and C++ compiler as part of the install. The gcc and g++ that come as rpms with most Linux distributions are fine.

Network setup?

It is assumed that your hardware has a working network connection through which packages may be retrieved and from which services may contact your system. The full OSG CE package is ~700MB.

Firewall?

If your installation is inside a site firewall, has a host firewall (ipchains/iptables), or is using TCP wrappers (/etc/hosts.{allow,deny}), you'll need to review the Firewall section of this guide and arrange for appropriate Internet access.

Time synchronization (NTP)

Each system should be setup to support network time protocol. Lack of synchronization generally complicates troubleshooting and can cause problems with the security infrastructures evaluation of eg. proxy lifetimes.

Reverse name lookups (DNS)

For the middleware to correctly function both the forward and reverse lookups as configured through a local DNS service are required for the IP address of the system.

Previous OSG or Grid3 Installations

If there is a previous installation of the OSG or Grid3 environment or other Globus middleware, please stop the processes which are currently running. This includes the Globus Resource Information Service (GRIS), MonALISA and other services configured to start upon initialization on your system. Also, if you've sourced the $VDT_LOCATION/setup.[c]sh on your previous VDT install, just log out and log back in to clear out your environment.

More information is provided in the OSG Shutdown Guide.

Creation and setup of local user accounts for VOs for OSG

UNIX user accounts need to be created by the system administrator for the VOs. You will need to create at least one local user account (with the appropriate configuration) for each VO to which you wish to provide resources. The uid and name of the account can be locally determined. You will be asked to provide a mapping from local accounts to VOs later in the installation, the following default accounts are assumed for examples in this document.

The accounts are:

• cdf ( cdfosg 783715 )
• fermilab ( fermilab 825662 )
• grase ( grase 783716 )
• fmri ( fmriosg 783717 )
• gadu ( gadu 783718 )
• mis ( misosg 803968 )
• sdss ( sdss 751566 )
• ivdgl ( ivdgl 751565 )
• star ( starosg 789088 )
• usatlas1 ( usatlas 751564 )
• usatlas2 ( usatlasa 789089 )
• usatlas3 ( usatlasb 789090 )
• usatlas4 ( usatlasc 55620 )
• uscms01 ( uscmsa 751562 )
• uscms02 ( uscmsb 751563 )
• ligo ( ligo 825671 )
• sam ( sam 825663 )
• samgrid ( samgrid 825664 )
• dosar ( dosar 825665 )
• des ( des 825666 )
• glow ( glow 825668 )
• grow ( grow 825669 )
• gridex ( gridex 825670 )
• nanohub ( nanohub 825672 )
• geant4 ( geant4 825673 )
• i2u2 ( i2u2 825674 )

In addition, you need a globus user account ( globus 825675 ) for web services to run.

Do you have a user certificate?

You will need a personal grid certificate to run validation tests. If you don't have a personal certificate, or don't know how to generate a grid proxy (grid-proxy-init, etc), contact your VO Support Center.

CE Site Adminstrator Overview

This section is intended to describe the information you will need available to configure your OSG CE node. It will also provide some general guidelines for your physical configuration of hosts and file systems.

You will be prompted for this information by the OSG_LOCATION/monitoring/configure-osg.sh script after the software is installed.

---

General Information

OSG Group, Resource Name, Sponsors, Policy URL

OSG Group This identifies the monitoring group that your site is participating in. Values:

• OSG - For Production
• OSG-ITB - For the integration testbed

Resource Name

Each OSG resource needs a unique name by which services and resources may refer to the resource. This name will be displayed on the Site Catalog and used in tables for other monitoring and accounting. For example, the site at University of New Mexico has a unique name of UNM_HPC; the site at University of Buffalo is Buffalo_CCR.

Note: The default name in most configuration scripts is the hostname of the node the installation is performed on. Do not use this name.

It is important that system administrators use the same Resource Name in all locations including the registration step, this will allow the Grid Operations Center (GOC) to validate monitoring systems and other grid services.

This unique name may be requested at several points during the installation process and may be referred to as:

• resource name
• site name
• farmname (in MonALISA)

Currently the complete list of Resource Names from the OSG Registration DB is the authoritative list of names. VO Resource Selector (VORS) and GridCat systems also contain a list of current resource names across both the OSG and ITB, and can be used to determine names already in use.

For a list of the resource names already in use, you may view:

• Listing Pulled from OSG Registration DB
• OSG Resource Names via VORS
• ITB Resource Names via VORS

• OSG Resource Names via GridCat
• ITB Resource Names via GridCat

Sponsors: The VO sponsors for your site. For example: usatlas, ivdgl, ligo, uscms, sdss... You must express the percentage of sponsorship using the following notation.

myvo:50 yourvo:10 anothervo:20 local:20

Policy URL: This is the URL for the document describing the usage policy/agreement for this resource.

---

Contact Information

Contact name: The site administrator's full name.
Contact email: The site adminstrator's email address.

---

Server location

City: The city your server is located in or near.
Country: The country your server is located in.
Longitude/Latitude: For your city. This will determine your placement on any world maps used for monitoring. You can find some approximate values for your geographic location from:

   http://geotags.com/
      or 
   you can search your location on Google

For USA:

• latitude is about 29 (South) ... 48 (North)
• longitude is about -123 (West coast) ... -71 (East coast)

---

Local Storage Information

These variables define various storage locations available on your site for jobs. The are described in detail in the a separate Local Storage Configuration document.

GRID, APP DATA, WN_TMP

GRID: Location where the OSG WN Client (wn-client.pacman) has been installed.

APP: Typically used to store the applications which will run on this gatekeeper.

DATA: Typically used to hold output from jobs while it is staged out to a Storage Element.

WN_TMP: Used to hold input and output from jobs on a worker node where the application is executing.

Storage Element (SE), SITE_READ, SITE_WRITE

Storage Element (SE): This is the Storage Element (SE) that is visible from all worker nodes accessible via this server (CE). It may be a SE local or close to the CE that is preferred as destination SE if the job does not have other preferences.

SITE_READ: Used to stage-in input for jobs using a Storage Element or for persistent storage between jobs. It may be the mount point of a dCache SE accessed read-only using dcap.

SITE_WRITE: Used to store to a Storage Element output from jobs or forpersistent storage between jobs. It may be the mount point of a dCache SE accessed write-only using dcap.

---

MonALISA monitoring information

This information will only be required if you intend to activate MonALISA TOC STARTINCLUDE ">MonALISA.

Ganglia host and port

If ganglia is being used to monitor you cluster, you will be asked for these attributes:

Ganglia host: The host machine ganglia is running on.
Ganglia port: The host machine's port ganglia is using.

VO Modules

If 'y', this will activate the VO Modules module in the MonALISA TOC STARTINCLUDE ">MonALISA configuration file.

---

Batch Queue Manager Information

The supported batch managers are:

• Condor
• PBS
• LSF
• SGE

For Condor: The CONDOR_CONFIG variable value is needed.
For SGE: The SGE_ROOT variable value is needed

Installing and Setting Up Pacman

Pacman is a package management tool that allows you to define, install, and configure software. The homepage of Pacman is http://physics.bu.edu/pacman/.

Pacman version 3.16.1 should be used for the OSG 0.4.1 installations.

Download

First, cd to an install directory somewhere. Then:
> wget http://physics.bu.edu/pacman/sample_cache/tarballs/pacman-3.16.1.tar.gz

Unpack and Setup

Then unpack the tarball:

> tar --no-same-owner -xzvf pacman-3.16.1.tar.gz

Next, setup your environment:
> cd pacman-3.16.1

For sh and bash shells:

$ source setup.sh

For csh and tcsh shells:
% source setup.csh

Installing OSG software using Pacman

These are general instructions for installing software from a Pacman cache.
> mkdir INSTALLATION_DIRECTORY  (e.g. /usr/local/osg)
> cd INSTALLATION_DIRECTORY
> pacman -get OSG:PACKAGE_NAME

Unsupported Platforms

If you get an error message indicating ".....is not yet supported", you may need to specify the pacman command with the following argument:
> pacman -pretend-platform:RHEL-3 -get OSG:ce
where you could substitute any platform for RHEL-3.

To see a list of the defined pacman platforms (not all of which are yet supported), execute:
> pacman  -platforms

setup.sh issues

Pacman normally removes the setup.sh file at the start of an installation and, if successful, restores the file. The VDT grid middleware will fail to work if the setup.sh script is unavailable.

To allow out of date setup scripts to remain during installation or updates execute the command:

> pacman -allow save-setup

To attempt recovery of the setup.sh file execute the command:
> pacman -setup

Removing packages with pacman

If you installed a package that you may later have deemed unnecessary, you can remove it by issuing the pacman command with the remove option.
> pacman -remove package_name

For details on these commands, see the Pacman home page.

Installing the OSG Worker Node Client Package

This must be installed in a location visible to jobs executing on the worker node. Refer to the Worker Node Client Guide for additional information.

Installing OSG CE Services

Now you're ready to start installing the Compute Element (CE) services. We distinguish between software that must be installed on the server/gatekeeper (eg., ce.pacman), and software which must be executable from the worker nodes.

This section addresses the software executed on the CE (server/gatekeeper node).

Choose an installation directory

This is typically something like /usr/local/grid, but whatever suits the local resource structure is fine. This directory does not need to be shared to the worker nodes. Any software which needs to be seen from the worker nodes may be provided by the worker node client package as described above.

For example (shown for sh, bash):

> export VDT_LOCATION=/usr/local/grid
> cd $VDT_LOCATION

Installing the OSG CE software package

The installation described here is done as root. Even though services will not run as root:

• Condor, MonaLisa^? and the GRIS will run as the user "daemon".
• Verify that the umask is set to "0022" prior to installation. Failure to do so may render the installation unusable.
• A few questions regarding trust of the caches from which the software is downloaded will be displayed.
  Please answer y (yes) so that the software can be retrieved.
• Finally, make sure there are no non-standard versions of perl, tcsh, or bash in your $PATH variable.

The following pacman commands will install the OSG software packages.

   >  pacman -get OSG:ce

See the Pacman section of this guide if you encounter an 'unsupported' platform message.

This will take between 10 and 60 minutes to complete, depending upon the system and network connection. During this time you may open a second terminal and watch the progress by monitoring the $VDT_LOCATION/vdt-install.log file.

You should not be asked any other questions during the installation process.

The installation should complete with the following message.

Downloading [srmclient-1.23.tar.gz] from [pacman.uits.indiana.edu]...
         6/6 MB downloaded...                               
Untarring [srmclient-1.23.tar.gz]...
Downloading [ml-patch.tar.gz] from [pacman.uits.indiana.edu]...
Untarring [ml-patch.tar.gz]...
 
Pacman Installation of OSG-0.4.1 Complete

Setup the OSG environment

Assuming the Pacman install completed without fatal errors, you should now be able to source the OSG setup environment. In your installation directory, from here forward known as the $VDT_LOCATION and source the setup script.

$ source setup.sh
         or
% source setup.csh

Optional packages for Condor, PBS, LSF, FBSNG, or SGE

An extra package may be required to setup access to an existing Condor, PBS, LSF, or SGE installation. Ensure that that the command-line utilities for your batch system are in your path, then install the appropriate package in your $VDT_LOCATION directory (for Condor, PBS, LSF, or SGE respectively):

> pacman -get OSG:Globus-Condor-Setup
> pacman -get OSG:Globus-PBS-Setup
> pacman -get OSG:Globus-LSF-Setup
> pacman -get OSG:Globus-SGE-Setup

For OSG-ITB substitute IVDGL for OSG:

> pacman -get iVDGL:Globus-Condor-Setup

and so forth.

This guide will not go into the detail on the installation of any of these optional packages.

Post install README

Read the $VDT_LOCATION/post-install/README file. We suggest that you read the file for information only, and follow the instructions in the various OSG installation guides presented in these pages, rather than those in the VDT README file.

Note: Do not be concerned with any error messages you may see in this README file. They are usually not a problem.

This page and others that it links to will take you through all the steps necessary to configure your installation.

The Managed Fork Queue

With OSG 0.4.1 an optional service called the Managed Fork has been introduced. This service replaces the default jomanager fork with a jobmanager which uses Condor to manage the incoming jobmanager fork requests.

Please note that the managed fork will only work in Condor 6.7.15 or greater; older versions of Condor will run all local universe jobs as soon as they are submitted or may result in an error.

The standard fork jobmanager is used to spawn jobs on the gatekeeper/headnode, the Managed Fork updates the fork jobmanager to submit requests to the local universe Condor queue instead. These jobs are still run on the gatekeeper/headnode, but the jobs may now be subject to limitations and priority as assigned by the local system administrator.

By using the "local universe" fork jobs are not actually subject to scheduling delays, unless the total number of allowed fork jobs of a given type is exceeded.

Managed Fork Package Installation

The installation is done in same directory as the OSG CE software. To specify an external Condor installation set the following two environment variables before installing:

• VDTSETUP_CONDOR_LOCATION: the location of your Condor installation (e.g. /opt/condor). The Condor bin/, sbin/, etc/, lib/... directories should be directly under this location.

• VDTSETUP_CONDOR_CONFIG (optional): The location of your Condor configuration file (if non-standard). Default is ${VDTSETUP_CONDOR_LOCATION}/etc/condor_config.

Otherwise a full Condor installation will be installed with the package.

cd $VDT_LOCATION

source $VDT_LOCATION/setup.sh
pacman -get OSG:ManagedFork

Enabling Managed Fork

To configure the default jobmanager to be the managed fork jobmanager execute the following command.

source $VDT_LOCATION/setup.sh

$VDT_LOCATION/vdt/setup/configure_globus_gatekeeper --managed-fork y

By default, the managed fork jobmanager will behave just like the fork jobmanager. If you wish to restrict it you need to modify your local Condor configuration. If you're using Condor from the VDT this can be done by editing $VDT_LOCATION/condor/local.<hostname>/condor_config.local.

Only allow 20 local universe jobs to execute concurrently:

START_LOCAL_UNIVERSE = TotalLocalJobsRunning < 20

Set a hard limit on most jobs, but always let grid monitor jobs run (strongly recommended):

START_LOCAL_UNIVERSE = TotalLocalJobsRunning < 20 || GridMonitorJob == TRUE

Disabling Managed Fork

To replace the default fork jobmanager execute the following command.

source $VDT_LOCATION/setup.sh
$VDT_LOCATION/vdt/setup/configure_globus_gatekeeper --managed-fork n

Further Details on Managed Fork

For more details on setup and configuration we refer to the VDT 1.3.10 release notes on managed fork.

Configuring the Public Key Infrastructure

Minimally, you will need a host certificate and private key to identify of your CE.

Setup and maintenance of Certificate Authority connection

Configure the DOEGrids Certificate Authority (CA) to be used by default by running the utility below.

If you are given the option "choose the option which matches "1c3f2ca8 : ...",

then enter "q" at the prompts.

  > $VDT_LOCATION/vdt/setup/setup-cert-request
  Reading from /g3dev/globus/TRUSTED_CA
  Using hash: 1c3f2ca8
  Setting up grid-cert-request
  Running grid-security-config...
  ......

A list of CAs were added as authorized CAs on your system.

Important Note: In the past, you had the option of putting these certificates in the

/etc/grid-security/certificates directory or in the local $VDT_LOCATION/globus/TRUSTED_CA directory. This is no longer an option. The OSG installation is pre-configured to place the certificates in the local TRUSTED_CA directory. The edg-crl-upgrade daemon will be updating CRLs in this local directory only. If you want to maintain certificates in the /etc/grid-security/certificates directory you should link it to the local TRUSTED_CA location (symlink in either direction) and copy the CA files appropriately.

Please review the list of authorized CAs and modify the set in $X509_CERT_DIR= $VDT_LOCATION/globus/TRUSTED_CA as needed to match your local policy.

The daemon edg-crl-upgrade should be running at all times in the background to refresh the CRLs from these CAs. If CRLs are not kept current, incoming connections will fail.

To check that it's running:

> ps axwww | grep edg-crl-upgrade

If not running, do

> /etc/init.d/edg-crl-upgraded start

The Certificate Scripts Package Guide which has been installed can assist you with choosing Certificate Authorities to trust and and periodically checking that the CRLs (Certificate Revocation Lists) have not expired.

Request and Install Host Certificate

After the pacman installation of the OSG CE software completes you should have the PPDG-Cert-Scripts package installed (in $VDT_LOCATION/cert-scripts) and the instructions below show the use of those scripts.

To authorize this resource for use, a host certificate needs to be obtained from an appropriate Certificate Authority.

• Currently the DOEGrids CA is the most common CA in use for OSG participants.

• The iVDGL and PPDG project instructions for getting a certificate are available on the iVDGL RA and PPDG RA sites (RA is Registration Authority).

Here is a brief guide to the process assuming you have a valid User Certificate.

Run the cert-request script to generate your host's private key (hostkey.pem) and submit the certificate request.

   $ cd $VDT_LOCATION
   $ source  ./setup.sh
   $ cert-request -ou s -dir .
   Processing OU=Services request.
   Give reason (1 line) you qualify for certificate, such as
     member of CMS experiment    or
     collaborating with Condor team,  etc.
    reason: testing for PPDG
   input full hostname: goofy.looney.tunes
   Generating a 2048 bit RSA private key
.   ....................................+++
    ..................+++
    writing new private key to './5842key.pem'
    -----
    input your email address: address@your.email.server
    input your complete phone number: 9995551212

   Choose a registration authority to which you are affiliated.
   _Enter__this____for this registration authority
        anl     ANL: Argonne National Lab
        epa     NCC-EPA: Environmental Protection Agency
        esg     ESG: Earth System Grid
        esnet   ESnet: DOE Science network
        fnal    FNAL: Fermilab host and service certificates
        ivdgl   iVDGL:  see www.ivdgl.org
        lbl     LBNL: Berkeley Lab
        lcg     LCG: LHC Computing Grid
        nersc   NERSC: computer center, see www.nersc.gov
        o       Other: if you can not tell what to choose
        ornl    ORNL: Oak Ridge National Lab
        pnnl    PNNL: Pacific Northwest National Lab
        ppdg    PPDG: includes BNL, JLab, SLAC and many HEP & NP experiments
   (choose from left column): ppdg
   ppdg
   PPDG
   You must agree to abide by the DOEGrids policies,
   at http://www.doegrids.org/Docs/CP-CPS.pdf
   Do you agree (y,N): y

   Your Certificate Request has been successfully submitted
   Your Certificate Request id: 2005

        You will receive a notification email from the CA when your certificate
        has been issued. Please disregard the instructions to download your
        certificate though a web browser and use the cert-retrieve script instead.

After the certificate is approved you will receive an email which includes the serial number of the new certificate. Use that serial number to retrieve the certificate and move it to the installation directory.

  > cert-retrieve -dir . -certnum 0x299
   using CA doegrids
   Checking that the usercert and ./5842key.pem match
   writing RSA key
   ./usercert.pem and ./userkey.pem now contain your Globus credential

   > mv ./usercert.pem /etc/grid-security/hostcert.pem
   > mv ./userkey.pem /etc/grid-security/hostkey.pem
   > chmod 444 /etc/grid-security/hostcert.pem
   > chmod 400 /etc/grid-security/hostkey.pem

The following command will verify the certificate is readable. The output shown will be similar, but specific to your request.

   > openssl x509 -text -noout -in /etc/grid-security/hostcert.pem

   Certificate:
       Data:
           Version: 3 (0x2)
           Serial Number: 665 (0x299)
           Signature Algorithm: sha1WithRSAEncryption
          Issuer: DC=org, DC=DOEGrids, OU=Certificate Authorities, CN=DOEGrids CA 1
          Validity
              Not Before: Dec 13 23:55:14 2005 GMT
              Not After : Dec 13 23:55:14 2006 GMT
          Subject: DC=org, DC=doegrids, OU=Services, CN=goofey.looney.tunes
          .........
.

Get an LDAP service certificate to run authenticated MDS (optional)

MDS is the monitoring and directory service. The GRIS (Grid Resource info service, a sensor that collects site data and publishes it to MDS) is configured by default to allow anonymous access only. Only those sites which keep anonymous access available will be able to publish their information to the OSG BDII.

Instructions are in the $VDT_LOCATION/post-install/README on how to allow only authenticated and authorized access. Authenticated access requires an LDAP service certificate (two files), owned by the MDS owner (if you installed as root, the owner is probably daemon -- check).

   $ source $VDT_LOCATION/setup.sh
   $ cert-request -ou s -dir . -host hostname.domain.tld -service ldap

And follow the instructions, which are very similar to the host cert instructions, except that you are creating ldapkey.pem and ldapcert.pem.

Configuring Globus and Your Job Manager

Globus Configuration

Globus has been pre-configured for this installation. If you wish reconfigure it, please review the VDT Globus Configuration Script document on using the $VDT_LOCATION/vdt/setup/configure_globus_gatekeeper script.

Review the configuration files /etc/xinetd.d/globus-gatekeeper and /etc/xinetd.d/gsiftp (or in /etc/inetd.conf) that were created during the pacman installation. Additionally, /etc/services file was updated with the following entries:

  
  gsiftp                  2811/tcp       # Added by the VDT
  globus-gatekeeper       2119/tcp       # Added by the VDT

If you are satisfied with this configuration, restart the xinetd (or inetd) daemon to pick up the configuration changes:

  /etc/rc.d/init.d/xinetd restart
  Stopping xinetd:                       [  OK  ]
  Starting xinetd:                       [  OK  ]

To verify that the gatekeeper is running at this point, you should be able to telnet to the public IP address of your site on port 2119 and get a response. telnet _hostname port_

It should return Connected to.... The same should be true of the gsiftp port (2811 by default).

Job Policy and the Batch Queuing System

The configuration of a production queue manager is beyond the scope of this document. Since there will be periodic validation submitted by DNs in the "mis" VO, it is recommended sites provide at least two levels of priority and assign the lowest priority to user mapped to the "mis" VO.

Configuring the OSG Attributes

At this point, there are several OSG attributes (environmental variables) that need to be established for required and optional CE services to run effectively.

These attributes are collected by executing the $VDT_LOCATION/monitoring/configure-osg.sh script. This script functions in question and answer mode and does the following:

1. Collects all the information described in the CE Site Administrator Overview section of this document.

2. It configure MonALISA using the $VDT_LOCATION/vdt/setup/configure_monalisa script.
   If you asked for MonALISA to be activated, an /etc/init.d/MLD will be added to your systems rc.d services and started.
   If you did not wish MonALISA to be activated, it will still run the configure_monalisa script but without the option to activate it.

3. On the first executionn only, it will copy the ./monitoring/locations/grid3-locations.txt to OSG_APP/etc with 666 permissions.

With each successful execution of the configure-osg.sh script:

• the osg-attributes.conf file is updated
• a backup of your previous osg-attributes.conf is saved as osg-attributes.conf.osgsave.[n]

The configure-osg.sh script is in a state of transition at this point. For the long term, the intent is to use this single script for the collection and distribution of all data needed on an OSG CE node to insure consistent and timely distribution of information and configuration. At this time, however, the configure_monalisa script is the only one currently included. The others that are candidates for inclusion are:

• configure_gip
• configure-misci
• configure_mds

Usage:

./configure-osg.sh             Executes the script in question and answer mode
./configure-osg.sh  --help     Steps through the set of information to be collected
./configure-osg.sh  --display  Displays the current osg-attributes.conf attributes

Simple Test of the CE Node (using local grid mapfile)

The first step is to configure the CE to allow access using your own Grid credentials. Once that's done you'll be able to run local tests and verify operation of your CE locally.

Make sure you have a grid proxy for yourself. (this is based on your certificate):

   > source VDT_LOCATION/setup.(c)sh
   > grid-proxy-init
    (you will be prompted for your GRID pass phrase)
 

Then, to get the subject (DN) of your proxy, run:

   >  grid-proxy-info -subject
Output....
   /DC=gov/DC=fnal/O=Fermilab/OU=People/CN=Dane Skow/UID=dane

Take the subject string and pre-pend it to the /etc/grid-security/grid-mapfile and assign it to a local user account (you can use any of the VO accounts you've created at the beginning to test). So the grid-mapfile should have at least one entry like:

 > cat /etc/grid-security/grid-mapfile
"/DC=gov/DC=fnal/O=Fermilab/OU=People/CN=Dane Skow/UID=dane" usatlas1

This is enough to enable the rest of the installation testing.

Simple Test of Fork Queue

Using the grid proxy you just created, you can test the Globus gatekeeper fork queue by executing:

     >  globus-job-run $(hostname):2119/jobmanager-fork  /usr/bin/id 
Output....
     uid=9872(usatlas1) gid=9872(usatlas1) groups=9872(usatlas1)

You should see the UNIX user account assigned based on the gridmap-file you created previously..

You can also view the $VDT_LOCATION/globus/var/globus-gatekeeper.log file to see the authorization messages:

  TIME: Sun Jan  8 23:12:43 2006
   PID: 29703 -- Notice: 6: Got connection 131.225.207.100 at Sun Jan  8 23:12:43 2006
  TIME: Sun Jan  8 23:12:43 2006
   PID: 29703 -- Notice: 5: Authenticated globus user:    /DC=gov/DC=fnal/O=Fermilab/OU=People/CN=Dane Skow/UID=dane

  TIME: Sun Jan  8 23:12:43 2006
   PID: 29703 -- Notice: 0: GRID_SECURITY_HTTP_BODY_FD=7
  TIME: Sun Jan  8 23:12:43 2006
   PID: 29703 -- Notice: 5: Requested service: jobmanager-fork
  TIME: Sun Jan  8 23:12:43 2006
   PID: 29703 -- Notice: 5: Authorized as local user: usatlas1
  TIME: Sun Jan  8 23:12:43 2006
   PID: 29703 -- Notice: 5: Authorized as local uid: 9872
  TIME: Sun Jan  8 23:12:43 2006
   PID: 29703 -- Notice: 5:           and local gid: 9872

  TIME: Sun Jan  8 23:12:43 2006
   PID: 29703 -- Notice: 0: executing /storage/local/data1/osg/globus/libexec/globus-job-manager
  TIME: Sun Jan  8 23:12:43 2006
   PID: 29703 -- Notice: 0: GATEKEEPER_JM_ID 2006-01-08.23:12:43.0000029703.0000000000 for /DC=gov/DC=fnal/O=Fermilab/OU=People/CN=Dane Skow/UID=dane on 131.225.207.100
  TIME: Sun Jan  8 23:12:43 2006
   PID: 29703 -- Notice: 0: GRID_SECURITY_CONTEXT_FD=10
  TIME: Sun Jan  8 23:12:43 2006
   PID: 29703 -- Notice: 0: Child 29808 started

Simple Test Of The Job Manager Queue

Now you can test the Globus gatekeeper job submissions for the job manager queue you are using. The example below is for the Condor job manager.

     >  globus-job-run $(hostname):2119/jobmanager-condor /usr/bin/id 
Output....
     uid=9872(usatlas1) gid=9872(usatlas1) groups=9872(usatlas1)

You should see the same results as with the fork queue submission.

The globus-gatekeeper.log file will look identical except for the "Request service" message:

   PID: 29703 -- Notice: 5: Requested service: jobmanager-condor

Simple Test Of GSIFTP Services

A simple test of the gsiftp services requires creating a simple file and then copying it from one location on your machine to the default storage element available for your CE. When you configured your OSG attributes (Configuring the OSG Attributes section of this document), you defined a default SE as a shared storage space with read-write access for all users. We will use this as the destination directory for the file we are copying.

Create a temporary file to be copied:

   $ echo "My test gsiftp file" > /tmp/gsiftp.test

Copy the file to the $OSG_DATA directory:

     > source $VDT_LOCATION/monitoring/osg-attributes.conf    (This is simply to get the OSG_DATA variable)
     $ globus-url-copy file:/tmp/gsiftp.test gsiftp://$(hostname)${OSG_DATA}/gsiftp.test

Verify the file was copied to the $OSG_DATA directory:

    > ls -l  $OSG_DATA/gsiftp.test
    -rw-r--r--  1 usatlas1 usatlas1  20 Jan  9 13:29 /storage/local/data1/osg/OSG.DIRS/data/gsiftp.test

To verify the accounting information was captured, you can view the $VDT_LOCATION/globus/var/log/gridftp.log for the ftp copy operation you just performed. You should see an entry similar to this for the transfer statistics collected by various systems like MonALISA and ACDC::

   DATE=20060110211714.329666 HOST=cmssrv09.fnal.gov PROG=globus-gridftp-server NL.EVNT=FTP_INFO START=20060110211714.247555 USER=ivdgl FILE=/tmp/gridcat-gsiftp-test.gridcat.21602.remote BUFFER=0 BLOCK=262144 NBYTES=28 VOLUME=/ STREAMS=1 STRIPES=1 DEST=[129.79.4.64] TYPE=RETR CODE=226

The authorization and other informational messages are captured in the $VDT_LOCATION/globus/var/log/gridftp-auth.log:

   [2834] Tue Jan 10 15:21:37 2006 :: Server started in inetd mode.
   [2834] Tue Jan 10 15:21:37 2006 :: Configuration read from /storage/local/data1/osg/globus/etc/gridftp.conf
   [2834] Tue Jan 10 15:21:37 2006 :: New connection from: cmssrv09.fnal.gov:38376
   [2834] Tue Jan 10 15:21:38 2006 :: User uscms112 successfully authorized
   [2834] Tue Jan 10 15:21:38 2006 :: Starting to transfer "/storage/local/data1/osg/OSG.DIRS/app/weigand.jgw.sh.2707".
   [2834] Tue Jan 10 15:21:38 2006 :: Finished transferring "/storage/local/data1/osg/OSG.DIRS/app/weigand.jgw.sh.2707".
   [2834] Tue Jan 10 15:21:38 2006 :: Closed connection from cmssrv09.fnal.gov:38376

Monitoring Setup

Configure MIS-CI

The Monitoring and Information Services Core Infrastructure (MIS-CI) provides information on the site environment and computing resources. The OSG-CE package includes MIS-CI. This section describes how to configure MIS-CI if you wish to enable it.

The $VDT_LOCATION/MIS-CI/configure-misci.sh script performs the configuration. It creates or adds a crontab for the MIS-CI information collectors. The Unix account for the MIS-CI scripts should be mis. By default, the script assumes the GridCat DN is mapped to the ivdgl user. You will need to use the --choose_user option to change to mis.

   $  cd $VDT_LOCATION
   $  SOURCE  ./setup.sh
   $  $VDT_LOCATION/MIS-CI/configure-misci.sh --choose_user
       Editing site configuration...
       Creating MIS-CI.db
           :
         ( a lot of information on the tables it is creating will appear before any questions are asked)
            :
       Would you like to set up MIS-CI cron now? (y/n) y
       At what frequency (in minutes) would you like to run MIS-CI ? [10] 10
       Under which account the cron should run ? [ivdgl] mis

       Frequency 10
       User mis
       Would you like to create MIS-CI crontab  ? (y/n) y
       Updating crontab
       Configuring MIS jobmanager
       /storage/local/data1/osg/MIS-CI/share/misci/globus/jobmanager-mis is created

       Your site configuration :
       sitename ITB_INSTALL_TEST
       dollarapp /storage/local/data1/osg/OSG.DIRS/app
       dollardat /storage/local/data1/osg/OSG.DIRS/data
       dollartmp /storage/local/data1/osg/OSG.DIRS/data
       dollarwnt /storage/local/data1/osg/OSG.DIRS/wn_tmp
       dollargrd /storage/local/data1/osg
       batcheS condor
       vouserS uscms01 ivdgl sdss usatlas1 cdf grase fmri gadu
       End of your site configuration


      If you would like to add more vo users,
      you should edit /storage/local/data1/osg/MIS-CI/etc/misci/mis-ci-site-info.cfg.
      You have additional batch managers :  condor .
      If you would like to add these,
      you should edit /storage/local/data1/osg/MIS-CI/etc/misci/mis-ci-site-info.cfg.

      configure--misci Done
      Please read /storage/local/data1/osg/MIS-CI/README

MIS-CI is collecting information using crontab as the user mis (or ivdgl if you left it as the default). Therefore, in order to stop MIS-CI processes, crontab should be removed. The script $VDT_LOCATION/MIS-CI/uninstall-misci.sh is provided for this purpose:

      >  cd $VDT_LOCATION
      >  source setup.(c)sh
      >  cd MIS-CI
      >  ./uninstall-misci.sh

After finishing configuring the MIS-CI, a few checks might be necessary:

1. Verify the crontab was created for the mis user.

         >  crontab -u mis -l
   

2. If you want to force an MIS-CI table update (due to fresh install or update), then as the MIS-CI user (mis), execute:

      >  $VDT_LOCATION/MIS-CI/sbin/run-mis-ci.sh
   

3. As a non-root user, verify that at least one table is filled.
   If you chose not to force an update, it might take 10 minutes or so before the tables are filled with current information.

       >   source $VDT_LOCATION/.setup.(c)sh
       >   grid-proxy-init
        (enter your password)
   
       >  globus-job-run <hostname>/jobmanager-mis /bin/sh siteinfo
         (Here <hostname> is the CE hostname.)
       ...... sample output ....
          id                       1 
         ymdt                     Wed Jan 11 19:00:01 UTC 2006 
         sitename                 ITB_INSTALL_TEST 
         hostname                 localhost 
         VOname                   local:100 
         appdir                   /storage/local/data1/osg/OSG.DIRS/app 
         datadir                  /storage/local/data1/osg/OSG.DIRS/data 
         tmpdir                   /storage/local/data1/osg/OSG.DIRS/data 
         wntmpdir                 /storage/local/data1/osg/OSG.DIRS/wn_tmp 
         grid3dir                 /storage/local/data1/osg 
         jobcon                    condor 
         utilcon                  fork 
         locpname1                 
         locpname2                 
         ncpurunning              0 
         ncpus                    4 
   

Configure MDS (GRIS)

The Globus information system is called MDS and is pre-configured to read the osg-attributes.conf information file.

The configuration script (VDT_LOCATION/vdt/setup/configure_mds) is executed automatically during the initial download with default values It also install the gris daemon as an /etc/rc.d service.

The gris daemon should have been started as a part of the initial installation. To verify:

>  ps -efwww |grep ldap
daemon    7584     1  0 15:25 ?        00:00:00 /bin/sh  /storage/local/data1/osg/globus/sbin/grid-info-soft-register 
                      -log  /storage/local/data1/osg/globus/var/grid-info-system.log 
                      -f  /storage/local/data1/osg/globus/etc/grid-info-resource-register.conf 
                      -- /storage/local/data1/osg/globus/libexec/grid-info-slapd 
                      -h ldap://0.0.0.0:2135 -d 0 
                      -f /storage/local/data1/osg/globus/etc/grid-info-slapd.conf
daemon    7627  7584  1 15:25 ?        00:00:00 /storage/local/data1/osg/globus/libexec/slapd 
                      -h ldap://0.0.0.0:2135 -d 0 -f /storage/local/data1/osg/globus/etc/grid-info-slapd.conf
daemon    7639     1  0 15:25 ?        00:00:00 /bin/sh  /storage/local/data1/osg/globus/sbin/grid-info-soft-register
                      -log /storage/local/data1/osg/globus/var/grid-info-system.log -register -t mdsreg2
                      -h cmssrv09.fnal.gov -p 2135 -period 600 
                     -dn Mds-Vo-Op-name=register, Mds-Vo-name=ITB_INSTALL_TEST, o=grid -daemon -t ldap 
                      -h cmssrv09.fnal.gov -p 2135 -ttl 1200 -r Mds-Vo-name=local, o=grid -T 20 -b ANONYM-ONLY 
                     -z 0 -m cachedump -period 30

If it is not running, you will need to restart it:

Usage:
   > /etc/init.d/gris  [start | stop ]

MDS should be configured for anonymous bind. You can send a test query to your local host which will perform no authentication on the user submitting the request . First, verify you have no proxy certificate (/tmp/x509u_(your_PID)). If one exists, remove it first. Then,

>  source $VDT_LOCATION/setup.sh

>  grid-info-search -anonymous
... your screen should scroll for a while showing a lot of data... 
....you can redirect the output to validate

Activate Your Site

The GridCat system maintains status and other information on all OSG sites as can be viewed here.

Sites added to GridCat are presumed to be inactive if a site state bit is not set to be 1 (see below).

• Inactive sites will have the site status dot with the grey color.
• Once the site becomes active, the site status dot will become either green or red, depending on the GridCat. test results.

Since the default site state is presumed to be inactive, the CE site administrator has to pro-actively switch the site state to be active. The activation is done by modifying the file $VDT_LOCATION/MIS-CI/etc/grid-site-state-info. and setting the value to 1 for the variable below:

export grid_site_state_bit =  1

NOTE: It might take up to 2 hours for registered sites to take effect in the GridCat display. If your site is not registered with the OSG-GOC see the instructions in the OSG Registration section of this document. Until your site is registered, it will not appear in GridCat

If your site decides to become inactive for various reasons, e.g., site maintenance, the site administrator should set the value of grid_site_state_bit to be other than 1.

Example grid-site-state-info file.

Optional Components

MonALISA TOC STARTINCLUDE ">MonALISA

To configure this optional component, see the MonALISA TOC STARTINCLUDE ">MonALISA document in this guide.

Generic Information Providers (GIP)

To configure this optional component, see the Generic Information Providers document in this guide.

Site Verification

site-verify

Now you're ready to run the full CE site verification test suite. All elements of this test should now pass.

Note: To run the site verify script you should not be root .

>   cd $VDT_LOCATION
>   source ./setup.sh
>   grid-proxy-init 
     ....enter your passphrase

>   cd verify
>   ./site_verify.pl 

The results will indicate the various tests that are performed with results indicating FAILED, UNTESTED, NOT WORKING, NONE or NO. conditions.

Authorizing Users: Operational Configuration

The earlier test case only authorized yourself as a local user. You should now go to Osg CE Authorization document and authorize other users before performing the OSG Registration of your service (otherwise no one but you will be able to access it!).

OSG Registration

To register the site with the OSG Grid Operations Center and into the Grid Catalog please use the webform located under the OSG Administrator Support page. If you are registering into the OSG, be sure to check the appropriate box for which grid catalog you are registering with. You should receive an email response automatically back from the GOC to the operations contact you supplied. If this response doesn't arrive within a reasonable delay, please resubmit your registration.

A minimal amount of information is needed for the OSG Grid Operations Center (GOC) to publish a site to the monitoring and operational infrastructure. This includes organization name, organization manager or designated representative name and email, security contact name and email, resource URL, support center, and form submitter.

While this minimal information will allow the GOC to publish your information to the monitoring tools, more information is requested to make site and support center communication easier. Please take time to fill out the form completely.

Firewalls

Grid components are distributed throughout the network, and services such as gatekeepers and data movement utilities are required to be accessible to the dynamic cloud of clients and peer services. This distributed and dynamic requirement places the burden of the security on the implementation of the application.

Due to the discovery of significant vulnerabilities in recent years, network-based applications are untrusted by default. To solve the application problem effort has focused on developing and deploying firewalls which restricts full and free network access. (You might say that this is analogous to building a house with no doors. Is it safe? Yes. Is it useful? No.)

Some essential network-based applications have been "hardened," such as mail relay services, web servers, and secure shell daemons. These are further protected further by IP address filtering to prevent access from unknown hosts or domains.

Grid components which are located behind network firewall face special challenges for Grid setup and operations.

There are two styles of firewalls usually encountered.

• A network firewall which is upstream from your server (usually centrally maintained). This blocks all traffic to your host.
• Host-based firewalls which are setup and maintained by individual host administrators. This is usually setup and configured by the iptables program which filters incoming network packets which arrive for the host.

In addition to host-based firewalls, hosts can choose to implement host based access rules (usually setup with the tcp_wrapper or hosts_allow utilities.

Network traffic can be blocked at the firewall for both incoming and outgoing dataflow depending on hostnames, ip addresses, ports and protocols.

A common setup is to allow any outgoing connection, while significantly (if not completely) restricting incoming connections. The Globus project provides thorough documentation, including but not limited to the Globus Toolkit firewall requirements document. Also remember that you may need to open ports on your firewall for your batch system (Condor, LSF, etc.!)

Information about Condor's use of ports

Port usage which may require firewall updates:

• Inbound:
  • MDS: 2135/tcp inbound

  • GRAM: 2119/tcp inbound
  • GridFTP: 2811/tcp inbound
  • GRAM callback: user-defined contiguous range (set via the environment variable GLOBUS_TCP_PORT_RANGE=beginport,endport). It should span at least 100 ports for a small site.
  • Monalisa: 9000/udp (for ABping measurements). These are specified in the file $VDT_LOCATION/MonaLisa/Service/VDTFarm/ml.properties

• Outbound:
  • GRAM callback: user-defined contiguous range (set via the environment variable GLOBUS_TCP_SOURCE_RANGE=beginport,endport).

GRAM and GridFTP need to know the port range that you've opened up via environment variables. You need to set the GLOBUS_TCP_PORT_RANGE=beginport,endport for inbound ports. If you restrict outbound connections, you will also need to set GLOBUS_TCP_SOURCE_RANGE=beginport,endport. These may be set either in $VDT_LOCATION/vdt/etc/vdt-local-setup.sh=, or in the xinetd configuration files -- the examples below use xinetd. The variables will be used by GRAM,

GridFTP, and any clients that require them.

The above ports and protocols must be open to and from all grid clients and server machines participating in the grid in order to provide minimal functionality.

In addition to the above, port 9443 must be open for both incoming and outgoing connections in order to test the web-services capabilities of the most recent versions of the VDT.

You also may need to open the following optional incoming ports for additional Grid services. Note that unlike the ones listed above, the following are optional and aare only needed if you are running those specific services or if required by your specific virtual organization.

• GIIS: 2136/tcp

• GSISSH: 22/tcp
• MyProxy: 7512/tcp
• VOMS: 8443/tcp
• RLS server: 39281/tcp

Here is a sample of the a /etc/hosts.allow with the GLOBUS services opened:

>  cat /etc/hosts.allow

  #
  # hosts.allow   This file describes the names of the hosts which are
  #               allowed to use the local INET services, as decided
  #               by the '/usr/sbin/tcpd' server.
  #
  sshd: 129.79.6.113 
  ALL : localhost
  vdt-run-gsiftp.sh : ALL
  vdt-run-globus-gatekeeper.sh : ALL

For RH9, RHEL3 or compatible iptables systems

The default firewall configuration for Red Hat's iptables sets the system up with a stateful packet filter. This is different than some legacy RH7 systems as by default no ports that are not explicitly opened by the iptables script will be open. This includes high numbered ports that are often used by grid services.

If your preference is to leave as much of the stateful packet filtering in place but enable just those grid services you want to deploy then you can use the following instructions.

Two changes need to be made to an OSG gateway with a host based iptables stateful firewall.

First is the configuration of the firewall itself. On RHEL or similar systems this is done in /etc/sysconfig/iptables

The Chain RH-Firewall-1-INPUT is a default chain for RHEL3. This chain is also sometimes called INPUT. Make sure the following rules use the chain that other rules in /etc/sysconfig/iptables do.

Note: For GSISSH this port is often already open for systems. You can use either this rule or the default rule setup at install time if you selected custom firewall and enabled ssh.

# Globus: Requires addition configuration in /etc/xinetd.d/globus-gatekeeper
# set: env = GLOBUS_TCP_PORT_RANGE=40000,50000
# This allows up to 10,000 ports and matches the globus config.
# How globus is configured to use these ports is subject to change in an upcoming
# release
-A RH-Firewall-1-INPUT  -m state --state NEW -p tcp -m tcp --dport 40000:50000 -j ACCEPT
# Monalisa, grabs 3 ports from the following range
-A RH-Firewall-1-INPUT  -m state --state NEW -p tcp -m tcp --dport 9000:9010 -j ACCEPT
# Gridftp
-A RH-Firewall-1-INPUT  -m state --state NEW -p tcp -m tcp --dport 2811 -j ACCEPT
# MDS
-A RH-Firewall-1-INPUT  -m state --state NEW -p tcp -m tcp --dport 2135 -j ACCEPT
# GRAM
-A RH-Firewall-1-INPUT  -m state --state NEW -p tcp -m tcp --dport 2119 -j ACCEPT

# Optional Services
# RLS Server
-A RH-Firewall-1-INPUT  -m state --state NEW -p tcp -m tcp --dport 39281 -j ACCEPT
# MyProxy
-A RH-Firewall-1-INPUT  -m state --state NEW -p tcp -m tcp --dport 7512 -j ACCEPT
# GSISSH/SSH
-A RH-Firewall-1-INPUT  -m state --state NEW -p tcp -m tcp --dport 22 -j ACCEPT
# GIIS
-A RH-Firewall-1-INPUT  -m state --state NEW -p tcp -m tcp --dport 2136 -j ACCEPT
# GUMS/VOMS
-A RH-Firewall-1-INPUT  -m state --state NEW -p tcp -m tcp --dport 8443 -j ACCEPT

Second, we configure Globus to use the allowed inbound port range.

/etc/xinetd.d/globus-gatekeeper

  service globus-gatekeeper
  {
         socket_type = stream
         protocol = tcp
         wait = no
         user = root
         instances = UNLIMITED
         cps = 400 10
         server = $VDT_LOCATION/vdt/sbin/vdt-run-globus-gatekeeper.sh
         env    = GLOBUS_TCP_PORT_RANGE=40000,50000
         disable = no
  }

If you restrict outbound connections (to the same port range, for example), you should also modify the gsiftp config file.

/etc/xinetd.d/globus-gsiftp

service gsiftp
{
         socket_type = stream
         protocol = tcp
         wait = no
         user = root
         instances = UNLIMITED
         cps = 400 10
         server = $VDT_LOCATION/vdt/sbin/vdt-run-gsiftp.sh
         env += GLOBUS_TCP_SOURCE_RANGE=40000,50000
         disable = no
}

Finally, add the port range(s) to $VDT_LOCATION/globus/etc/globus-job-manager.conf to ensure that it is picked up by other services by adding the following lines (omit the globus-tcp-source-range line if you do not restrict outbound connections):

        -globus-tcp-port-range 40000,50000
        -globus-tcp-source-range 40000,50000

Note: $VDT_LOCATION should be set by the pacman installer

If you limit the globus-related port range to certain values, it may be necessary to adjust the Linux ephemeral port range to avoid these values. If this has not already been done, check the /etc/sysctl.conf for the following lines and insert if needed:

# Limit ephemeral ports to avoid globus tcp port range
# See OSG CE install guide
net.ipv4.ip_local_port_range = 10240 39999

Save and exit if edited. Then, if you changed it, apply the changes by doing: sysctl -p

After editing the above files run the following commands

  # /etc/rc.d/init.d/iptables restart
  Flushing firewall rules:                                   [  OK  ]
  Setting chains to policy ACCEPT: filter                    [  OK  ]
  Unloading iptables modules:                                [  OK  ]
  Applying iptables firewall rules:                          [  OK  ]
  # /etc/rc.d/init.d/xinetd reload
  Reloading configuration:                                   [  OK  ] 
  

Troubleshooting Guide

As you install, monitor the $VDT_LOCATION/vdt-install.log.

• If pacman tries to retrieve something from a website that's having problems, you'll get an error message that's unrelated to the real problem because pacman can't recognize 404 errors when downloading tarballs. For example, when the PRIMA download site was down, it told us the file wasn't in the correct format:

vdt-untar is untarring prima-0.3.x86_rh_9.tar.gz
gzip: stdin: not in gzip format 

Shutdown Guide

Please see the OSG Shutdown Guide.

Major updates:

-- rquick@iupui.edu Company Name: UITS Company URL: http://uits.iu.edu Country: USA Comment: My Links TWIKIWEB .WelcomeGuest to learn TWiki Sandbox ...">RobQ - 01 May 2006
to top