Configuration and templates¶
The runOrca.py
and allocateNodes.py
utilities in the
ctrl_execute package use a template system that is similar to the Condor
plug-in that Orca uses. The ctrl_platform_lsst and ctrl_platform_gordon
contain platform specific information for running jobs on each of those
platforms. These are all discussed below. You should be familiar with the
concepts laid out in the explanation of the Condor plug in, available on this
page.
The command line arguments and config files contain values which are substituted for their corresponding $VAR type names in the template files when the utilities are run.
For example, the $HOME/.lsst/condor-info.py
is used to store
information about the user’s login name and home directory for each platform.
root.platform["gordon"].user.name = "juser"
root.platform["gordon"].user.home = "/home/juser"
root.platform["lsst"].user.name = "joeuser"
root.platform["lsst"].user.home = "/lsst/home/joeuser"
user.name
in the config file is$USER_NAME
in template filesuser.home
in the config file is$USER_HOME
in template files
If $USER_NAME
or $USER_HOME
is encountered anywhere in a template file,
the corresponding values are substituted there. For example, a file written
using template that looks like this:
cd $USER_HOME
echo "hello" > $USER_NAME
will end up looking like this for the “gordon” platform:
cd /home/juser
echo "hello" > juser
This substitution method is used throughout all the template files.
The values for these variables are noted below in the discussion of the config files.
allocateNodes.py¶
The allocateNodes.py command is the interface to the PBS submissions queue and is used to submit Condor glide-in requests to XSEDE nodes. The pex_config files and templates in the ctrl_platform directories are used to configure the PBS file submitted to XSEDE.
pbsConfig.py¶
The allocateNodes.py command executes PBS jobs to “glide-in” Condor nodes to the local cluster.
The file etc/config/pbsConfig.py
under the ctrl_platform_* packages
contains information to configure those PBS jobs. This file does not exist, and
is not needed in ctrl_platform_lsst, because we do have to allocate nodes to
the Condor cluster on that platform. They’re permanently allocated.
From ctrl_platform_gordon:
root.platform.queue = "normal"
root.platform.email = "#PBS -m be"
root.platform.scratchDirectory = "$USER_HOME/condor_scratch"
root.platform.loginHostName = "gordon.sdsc.edu"
root.platform.utilityPath = "/opt/torque/bin"
root.platform.queue
- the name of the PBS queue to submit jobs to. Template variable: $QUEUEroot.platform.email
- value to insert into the PBS file to send the user e-mail before and after jobs execute. Template variable:$EMAIL_NOTIFICATION
root.platform.scratchDirectory
- the directory on the REMOTE system to use for Condor scratch space. Template variable:$SCRATCH_DIR
root.platform.loginHostName
- the node to log into execute shell and copy commands. Template variable:$HOST_NAME
root.platform.utilityPath
- the REMOTE system’s directory where the PBS utilities (qsub, qdel, etc.) are located. Template variable:$UTILITY_PATH
generic.pbs.template
¶
This file generic.pbs.template in the etc/templates
directory of XSEDE
node ctrl_platform packages is the template that is filled out changed into
the PBS file which is used to submit glide-in requests to the PBS queue on the
target platform. IMPORTANT: The top part of this file looks like a comment
block, but is actually part of the PBS script. Do not delete that portion of the
file. It contains the “#PBS” statements which are used to configure the PBS
request.
Below that PBS statement block is a shell script that is executed once the nodes
are allocated. The $PBS_NODEFILE
referenced in this part of the PBS
file will contain a list the slots allocated by the PBS job. This part of the
script runs through the list of slots, and runs an SSH command which sets up a
condor-glidein to run:
ssh ${hostname[$num]} 'export CONDOR_CONFIG=$SCRATCH_DIR/$GENERATED_CONFIG;export _condor_CONDOR_HOST=lsst-launch.ncsa.illinois.edu;export _condor_GLIDEIN_HOST=lsst-launch.ncsa.illinois.edu;export _condor_LOCAL_DIR=$USER_HOME/condor_local;export _condor_SBIN=/oasis/projects/nsf/nsa101/srp/condor/condor-7.4.4-r1/sbin;export _condor_NUM_CPUS=$SLOTS;export _condor_UID_DOMAIN=ncsa.illinois.edu;export _condor_FILESYSTEM_DOMAIN=sdsc.edu;export _condor_MAIL=/bin/mail;export _condor_STARTD_NOCLAIM_SHUTDOWN=1800; /oasis/projects/nsf/nsa101/srp/condor/glidein/glidein_startup_gordon -dyn -f' &
CONDOR_CONFIG
points at theallocateNodes.py
generated Condor config file. This is not apex_config
style file._condor_CONDOR_HOST
points at the head node of the Condor flock_condor_GLIDEIN_HOST
points at the host configurated to accept glide-in requests to the Condor flock._condor_LOCAL_DIR
points at the Condor’s local directory for job execution._condor_SBIN
points at the directory containing the binaries used by Condor_condor_NUM_CPUS
assigns the number of slots per node_condor_UID_DOMAIN
assigns the network domain name of the Condor flock that will host the glide-in_condor_FILE_SYSTEM_DOMAIN
assigns the network domain name of the file system which the glide-in is coming from_condor_MAIL
assigns the mail program Condor will use to send mail messages_condor_STARTD_NOCLAIM_SHUTDOWN
assigns the time, in seconds, which a glide-in slot can be inactive before it is removed from the Condor flock. In other words, if no jobs are assigned to this slot within this time, it is removed from the Condor flock and is no longer available for jobs. If DAGman does not fill up the queue fast enough, a portion of a large glide-in request could be deallocated before any jobs are assigned.
glidein_condor_config.template
¶
This is the template for Condor config file which condor uses on startup. This
is a very simple Condor config file. The only additional non-standard value
used here is the NODE_SET
variable:
NODE_SET = "$NODE_SET"
Note the use of the double quotes around the value. Those are required. If you
forget those, NODE_SET
will not be set properly.
When the template is written, the real value of $NODE_SET
is
substituted. The Condor matching algorithm has this value available to match
jobs. When a job is submitted, it has a “Requirements” line like this (from the
preJob, postJob and worker Condor submission file): [{{ Requirements =
(FileSystemDomain? == “sdsc.edu”) && (Arch != “”) && (OpSys? != “”) && (Disk !=
-1) && (Memory != -1) && (DiskUsage? >= 0) && (NODE_SET == “$NODE_SET”) }}}
When Condor matches jobs to machines, it matches this line. Our custom addition
of NODE_SET
allows us to match particular jobs to particular sets of
machines. That’s how we keep two different sets of allocated machines and two
sets of jobs from using each other’s resources.
$HOME/.lsst/node-set.seq
¶
Stores the sequence number of the node-set value. This is incremented by the
allocateNodes.py
utility.
Variables available for templates when using allocateNodes.py¶
$USER_NAME
- remote login name$USER_HOME
- remote home directory$NODE_COUNT
- number of nodes requested$SLOTS
- number of Condor slots per node$WALL_CLOCK
- maximum time to allocate these nodes. This is in the form “HH:MM:SS”$QUEUE
- PBS queue to submit to$EMAIL_NOTIFICATION
- string to insert into the PBS file for e-mail notification of job execution$SCRATCH_DIR
- remote scratch directory$UTILITY_PATH
- remote directory where the PBS utilities are located.$NODE_SET
- the name of the node set for this glide-in$OUTPUT_LOG
- the name of the output log for the entire Condor job$ERROR_LOG
- the name of the error log for the entire Condor Job$GENERATED_CONFIG
- the name of the template file generated Condor config file.
Files used by runOrca.py¶
The runOrca.py command is an interface to the orca.py command available in the ctrl_orca package. It’s job is to configure templates for the Orca Condor plug-in, and then execute the Orca command on those resulting templates. Those templates specify how to construct a Condor DAG file, and what gets executed in a Condor “pre-job”, “worker”, and “post-job”.
Job Operating Environment¶
The runOrca.py takes the LSST stack environment at the time of execution on the launching platform, and duplicates it on the target platform. This is done in one of two ways, depending on the platform, and depending on the command line arguments you give it.
If you launch jobs to the LSST cluster, runOrca.py will set up the Condor jobs it launch to inherit your shell environment at the time you launch the job. That means that anything you have set up in the LSST stack will automatically be set up an used on the LSST cluster nodes when the jobs runs. We can do this because we’re running on a system with duplicate system software and a shared filesystem across the nodes. All versions are set up, including anything that set up in your local space (shown as LOCAL: in eups list). Job Operating Environment
If you launch jobs to an XSEDE node, runOrca.py will note all the versions of the software you’re running, and use a remote stack to set up those same versions. Note that no locally setup packages (shown as LOCAL: in the eups listing for that package) are setup remotely.
In order to speed up the worker tasks, there is an initial job (called the
preJob
) that sets everything up, and saves the environment into a shell
script called env.sh
. When a worker job starts, it executes that shell
script to set up the environment.
Because of these different operating environments, we need to set things up
differently in the configuration file that is given to Orca to execute. In the
ctrl_execute/etc/templates
directory, you’ll find the files:
config_with_getenv.py.template
config_with_setups.py.template
These templates are very similar. The differences are that they point to
different preJob
, worker
, and postJob
templates in the
ctrl_platform_* packages, the “setups” templates adds keywords for
NODE_SET
and CTRL_EXECUTE_SETUP_PACKAGES
.
The NODE_SET
variable is used to specify which Condor “node set” to
run on. See http://dev.lsstcorp.org/trac/wiki/Orchestration/SelfService for
more information about the node set concept. Note that while it is specified
here, if the Condor job is on the LSST cluster, this value is never used, since
it never appears in the ctrl_platform_lsst scripts.
The CTRL_EXECUTE_SETUP_PACKAGES
variable is used by Orca to
substitute the setup commands in the preJob script.
Config files in ctrl_platform packages¶
execConfig.py
¶
The names of directories were Condor jobs are executed, where data is located,
etc., are different from platform to platform, as you would expect. This
information is kept in the etc/config/execConfig.py
Config file, for
each platform.
From ctrl_platform_gordon:
root.platform.localScratch = "$HOME/condor_scratch"
root.platform.defaultRoot = "/oasis/scratch/$USER_NAME/temp_project/defaultRoot"
root.platform.dataDirectory = "/oasis/projects/nsf/nsa101/srp/inputdata/Sept2012/input"
root.platform.fileSystemDomain = "sdsc.edu"
root.platform.localScratch
- this is where Condor stores its files when you execute runOrca.py. Template variable: $LOCAL_SCRATCHroot.platform.defaultRoot
- this is the remote directory where orca jobs are executed. Output files and log files for the job appear here. Template variable:$DEFAULT_ROOT
root.platform.dataDirectory
- this is where the data that jobs use is kept. Template variable:$DATA_DIRECTORY
root.platform.fileSystemDomain
- this is the domain of the nodes of the remote cluster. Template variable:$FILE_SYSTEM_DOMAIN
Template files in ctrl_platform packages
Under the etc/template
directory you’ll find subdirectories containing
template files. Under ctrl_platform_lsst/etc/templates
you’ll find the
directories setups
and getenv
. Under
ctrl_platform_gordon/etc/templates
you’ll find “setups”. These are the
directories specified in the config_with_getenv.py.template
and
config_with_setups.py.template
files mentioned above. Each of those
directories contain the following six files:
postJob.condor.template
postJob.sh.template
preJob.condor.template
preJob.sh.template
workerJob.condor.template
worker.sh.template
The files with the condor.template suffix are all Condor templates that Orca
command will use to create “real” Condor submission files. For the most part,
these are standard Condor files. The main difference between these files in the
ctrl_platform packages is that the FileSystemDomain?
is set to the
platform’s network domain. For jobs that allocate remote nodes for glideins,
NODE_SET
is also specified here.
The files with the sh.template suffix are the scripts used during execution.
The preJob.sh.template and postJob.sh.template files are processed and turned
into .sh files which are executed once, and the worker.sh.template file is
turned into a worker.sh file which is executed as many times as specified by a
combination of the input file size and the number of ids per job that are
processed. This is all setup and used by a generated DAGman script. Note that
the setups/preJob.sh.template
contains code to handle setting up the
stack and saving the environment in the file env.sh
cd $EUPS_PATH
. $EUPS_PATH/eups/default/bin/setups.sh
setup lsst
cd $ORCA_DEFAULTROOT
cd $ORCA_RUNID
echo "Begin setups:"
date
$CTRL_EXECUTE_SETUP_PACKAGES
echo "Finished setups:"
date
This portion of the script sets up the stack, moves to the correct run
directory, and then executes the setups. The runOrca.py
command takes
the snapshot of the users EUPS environment and creates a series of setup -j
<pkgname>
references which it inserts in place of the
$CTRL_EXECUTE_SETUP_PACKAGES
variable. After everything has been setup, a
small portion of python code is executed which saves the environment.
As a reference, a processed preJob.sh.template
file would look something
like this, for the portion of code listed above:
cd /oasis/scratch/ux453102/temp_project/lsst/beta-0713/lsst_home
. /oasis/scratch/ux453102/temp_project/lsst/beta-0713/lsst_home/eups/default/bin/setups.sh
setup lsst
cd /oasis/scratch/srp/temp_project/defaultRoot
cd srp_2012_1004_163420
echo "Begin setups:"
date
setup -j cfitsio 3290+2
setup -j astrometry_net 0.30
setup -j sqlite 3.7.10+1
setup -j eigen 3.0.2+1
setup -j apr 1.3.3+1
setup -j ctrl_sched 5.1.0.0+16
setup -j datarel 5.2.1.5+4
[...setup lines deleted...]
setup -j gsl 1.15+1
setup -j astrometry_net_data sdss-2012-05-01-0
echo "Finished setups:"
date
The worker.sh.template
setups up the user’s environment, runs the EUPS
environment shell script env.sh
and processes the command line —id
arguments for the command given to runOrca.py, appending those id
arguments to the command itself. It also saves the resulting log file to a
subdirectory below the directory below “logs”. These log files are divided into
subdirectories based on the machine they ran on, and the Condor “slot” they ran
on, finally specified by the worker id number they were in the entire job.
Variables available for templates when using runOrca.py¶
$USER_NAME
- remote login name$USER_HOME
- remote home directory$DEFAULT_ROOT
- remote default root for job execution directories to be created.$LOCAL_SCRATCH
- local directory where files used by Condor are stored.$DATA_DIRECTORY
- remote directory where the data used for execution is stored.$IDS_PER_JOB
- the number of ids to assign per worker jobs$NODE_SET
- the name of the node set to match jobs with$INPUT_DATA_FILE
- the name of the input data file containing ids to process$FILE_SYSTEM_DOMAIN
- the network domain name of the remote file system$EUPS_PATH
- the directory containing the LSST stack used by EUPS$COMMAND
- the command passed into the runOrca.py job. This is the command that executes across all ids given in the input data file$CTRL_EXECUTE_SETUP_PACKAGES
- This is the listing of the setup -j <pkg> <version> commands of the LSST EUPS environment at the time runOrca.py is run.