Submiting Jobs

Often you will find yourself with the need to run batch jobs on Condor. This should be done entirely on the tier3's work/fast disks instead of on green.

Submission

To run a batch job using Condor, you need two things:

• an executable program or script that you want to run (make sure that the file is marked executable, use "chmod +x myjob.sh")
• a condor submit script

A submit script from the work disk T3work1 looks like this:

Universe = vanilla
Executable = myjob.sh

Arguments = myArgs

GetEnv = True

Input = /dev/null
Output = myjob.out.$(Cluster)
Error = myjob.err.$(Cluster)
Log = myjob.log.$(Cluster)

#Concurrency_Limits = DISK_T3WORK1:40

Queue

INPUT, OUTPUT, ERROR refer to UNIX STDIN, STDOUT and STDERR, e.g. the normal input and output stream from a terminal program. They do not refer to any other files that your program may open for reading or writing. You can pass arguments to your execution scripts using the Arguments line. GetEnv will use the same environment variables as they were present as the time of submission. $(Cluster) just contains the job number, if you don't use it, your output will always be in the same file, if you re-submit.

Concurrency_Limits will limit the number of jobs running off of one disk. This line is currently commented out (with the #) but if you have many jobs reading in from or out to one disk you will want this line to prevent limiting the DiskIO.

Ok, queue it (I've saved the above text into a file named myjob.submit):

condor_submit myjob.submit

You can see its status in the job queue using "condor_q".

Submitting Multiple Jobs

The above submit script has a limitation, if you run it multiple times, the output and error files will be overwritten by later jobs. If you have a set of jobs to run (running the same program multiple times) you probably want to create a different submit file for each. This might look like:

Universe = vanilla
Executable = myjob.sh

Argument = $(Process) $(Cluster)

Input = /dev/null
Output = myjob.out.$(Process)
Error = myjob.err.$(Process)
Log = myjob.log.$(Process)
#Concurrency_Limits = DISK_T3WORK1:40

Queue 25

A few elements are important if you want to submit multiple jobs:

• You need to add Queue 25 so that 25 subjobs (with the same job number) will be sent.
• You can use the $(Process) variable to keep track which sub job is running. The value will range from 0-24.
• The value of $(Process) will not be exposed directly to the job, so you need to pass this to the arguments of your script.

Disk IO

See also MSUTier3Storage.

Disk storage systems are able to perform a finite number of Input/Output operations per unit time. The specific limits depend on the disk system and the data access pattern (note that often the size of the dataset being read or written often isn't an issue). For hard disks, sequential IO (reading forward through a file) is much faster than random IO (reading from multiple files or random parts of a single file). If a number of jobs are requesting different files from a single storage system, the access pattern will be very random and performance can be low.

Given the number of job slots in the cluster, some jobs are able to overwhelm the performance of storage systems. This can result in increased runtime for a set of jobs and also can impact other users of the cluster. If you observe that the runtime of your jobs is larger than the CPU time, it is likely that the job is limited by storage performance.

Our Condor configuration allows users to specify a counter to limit the number of jobs running at once. A counter is setup for each of the NFS storage areas:

DISK_T3WORK1
DISK_T3WORK2
DISK_T3WORK3
DISK_T3WORK4
DISK_T3WORK5
DISK_T3WORK6
DISK_T3WORK7
DISK_T3WORK8
DISK_T3FAST1
DISK_T3FAST2
DISK_T3FAST3
DISK_T3FAST4
DISK_HOME_LIMIT

The counters are all set to a maximum of 10,000. To use this mechanism as for a allocation of the total in each job submit script. Condor will only run as many jobs as allowed by the sum of the counter. For instance, if you ask for "40" in each job, then up to 250 jobs will run at once. Note that the maximum is for the disk, so if other users have their concurrency limit set then it is the total of both users jobs that cannot exceed 10,000.

Concurrency_Limits = DISK_T3WORK1:40 

Requirements

Condor matches jobs to execution slots based on the requirements of the job and characteristics of the execution slot. This matchmaking method is a bit different than some other batch system that rely primarily on administrator defined queues.

Memory

For example, execution slots on the Tier3 are configured with 1GB, 2GB or 4GB of memory. If your job needs more than 1GB of memory, specifying a memory requirement will cause it to be run on an execution slot with more available memory.

Add this to the submit script before the queue statement (the units are MB):

Request_memory = 4000

To see all the characteristics of a given job slot, use a command like:

condor_status -long slot1@c-115-1.aglt2.org

To list the job slots that meet a given requirement use the -constraint option:

condor_status -constraint "memory > 4000"

Priorities

A limited number of job slots are setup to use for jobs that need to start quickly, even if normal resources are busy. There is no technical limit to using these slots, but please only use them for small numbers of jobs and/or in special circumstances. An example use is to submit a test job for debugging (remember to remove the requirement once testing is done).

Note that these are slots number 13 and 14 on 8 core nodes with Intel's Hyperthreading CPU feature --- no physical CPU cores are reserved for these slots.

To use this add the following to the job submit script (must be before the queue statement):

+MSU_Queue = "short"

Advanced topics

One-line submission script

Use the following script to submit in one line:

python /home/dta/scripts/condor_submit.py myscript.sh

Currently the script supports:

• submission of a simple script, creating output files in a directory "myscript.sh.condorlogs"
• mutiple jobs with -t option, setting ProcId to the subjob number
• mutiple jobs with complex patterns, such as -t 1-5,10-15 (useful if you want to repeat jobs 1-5 and 10-15)
• Use -h to see all the options

Interactive session on worker node

If you set up a key pair on the login machines (green, white, etc), you can use that key to log into cc-113-30.msulocal, which is one of the worker nodes.

Please use this power responsibly

It should only be used for small tests to investigate differences between how processes run on the login machines and how they are running on the condor queue.

Job information

There are some environment variables that are available in the running job. You can look into _CONDOR_JOB_AD which contains the location of a file that has the full information of the job, e.g. process number, job number, output file location, etc. Parsing this text file can give you information about the meta data of the job.

-- DucBaoTa - 20 Nov 2013