Home > User Support > Compute >Computing & Running Jobs Overview

Submitting and running jobs on TeraGrid computing resources requires an allocation. If you have not yet obtained an allocation for computing time, please refer to the following sections of this Web site:

• TeraGrid Resource Catalog for list of computation resources
• Access section for information on allocations, accounts, and Getting Started Guide

a wide range of tools and commands available for running jobs on the TeraGrid. The best tools for each job depend on your workflow needs. Several software layers exist to simplify job submission. For local job submissions, PBS (a portable batch system) can be used to create and submit batch jobs to a large number of cluster machines. For remote job submissions, the Globus Toolkit lays on top of PBS, but includes authentication, scheduling, and resources description tools required for the remote submission. However, if there are a number of jobs to be run independently, Condor-G provides a software layer on top of Globus, allowing advanced job submission and monitoring capabilities using a single script. In the case of a workflow pipeline, where jobs have dependencies, a Condor DAGMan script can control multiple Condor-G scripts enforcing order-of-execution dependencies.

In cases where a user wishes to distribute jobs across CPUs from multiple allocated resources, MyCluster constructs a personal Condor pool from the allocated resources. MyCluster creates an environment where jobs can be submitted using Condor commands or DAGMan pipelines, but hides the intricacies of maximizing computation throughput in a distributed computing infrastructure.

For details on other compute tools, see the individual pages:

Compilers Condor-G Globus MyCluster

MPICH-G2 Multi-site VMI PBS SoftEnv

Set Up

Before submitting a job you'll need to move your raw data files to the appropriate resource and to set up your programming environment. See the pages below:

• Transferring Files
• Programming Environment

Job Submission Requirements

TeraGrid sites require specifying certain parameters when submitting jobs to the queue. To select multiple types of nodes or multiple specific nodes in a single job, use these parameters:

host_types, host_names, host_xcount, and xcount

For example, specifiying (host_types=himem lomem)(host_xcount=16 8)(xcount=1 2) will select 16 himem (high memory) nodes with one process per node AND 8 lomem (low memory) nodes with two processes per node.

By site

Indiana | NCSA | PSC | Purdue | UC/ANL | NCAR

Indiana

IU requires a project id for all jobs on all resources that use the PBS job manager.

Remote Job Submission:
Globus 2.4.3 Gram JobManager

Local Job Submission
PBSPro 5.4.1
Condor-G
Globus 2.4.3 Gram JobManager

---

NCAR

For information on running batch jobs at NCAR, please see the NCAR Frost User Guide at http://www.cisl.ucar.edu/docs/frost/access.jsp#batch.

---

NCSA

Cobalt

Job submission to Cobalt requires the specification of a job a memory limit.
I.e., (-l mem=XXXXMB)

---

PSC

NOTE: Include project account numbers before entering queue.

Remote Job Submission:
Globus 2.4.3 Gram JobManager

Local Job Submission
OpenPBS
Condor-G
Globus 2.4.3 Gram JobManager

---

Purdue

Lear

For PBS clusters:

All jobs must have a project specified, either via "qsub -A project" (local submissions)
For globus submissions:

• Pre-WS GRAM RSL: (project=...)
• WS-GRAM XML <project>..</project>

Condor

All jobs must specify a project.
For globus submissions (jobmanager-condor):

• Pre-WS GRAM RSL: (project=...)
• WS-GRAM XML <project>..</project>

Non INTEL/LINUX architectures may be selected via GRAM with RSL like the following:

(condorsubmit=(Requirements = ( Arch=="Intel") && ( OpSys=="WINNT51" ) ))
(condorsubmit=(Requirements = ( Arch=="IA64") && ( OpSys=="LINUX" ) ))
(condorsubmit=(Requirements = ( Arch=="SUN4U") && ( OpSys=="SOLARIS" ) ))

Direct condor_submit requires a valid project number be specified in the submit file, as the value for the job ClassAd "TGProject". Condor will not start a job without a valid project. +TGProject = "TG-XY123456" is the format of this ClassAd.

Jobs in the Grid (condor-G), Scheduler (DAGman jobs), or Local Universes do not require TGProject to be set.

---

UC/ANL

Include project account numbers before entering queue
Explicitly specify compute nodes when specifying how many of them (with an additional :ia64-compute argument).

1) All jobs without a valid project for that user are rejected. A project can be selected thru ONE of the following methods:

1a) user specified in GT2 GRAM or GT4 pre-WS gram using "(project=...)"
1b) user specified in GT4 WS GRAM using "<project>...<project/>"
1c) selected automatically if the user only has one project
1d) selected by the user using the tgprojects command if (s)he has multiple projects

2) If the user cares what type of node they get (ia64, ia32, large frame buffer, standard frame buffer, 1.3 madison, 1.5 madison, etc) they must specify a required node type using ONE of:

2a) "(count=<n>:<property>)" in GT2 GRAM and GT4 pre-WS gram
2b) using the GT2 GRAM and GT4 pre-WS GRAM JobManager enhanced description
2c) using the GT4 WS GRAM PBS Node Selection extension 5.2.4. PBS Node Selection Parameters

Remote Job Submission:
Globus 2.4.3 Gram JobManager

Local Job Submission
Torque 1.1.0 (OpenPBS, PBSPro compatible)
Condor-G
Globus 2.4.3 Gram JobManager

Grid Tools

The Globus Toolkit and Condor-G tools provide the ability to submit jobs remotely from your local machine to the TeraGrid platforms.   You can also use these tools on each TeraGrid platform.

• Running Jobs Using Globus Commands
• Running Condor-G Jobs

Submitting a job against a TG project

This section contain information on specifying projects when submitting jobs to OpenPBS/Torque, Globus, and Condor-G. This information is relevant to all TG sites.

• OpenPBS/Torque
  For interactive jobs specify the "-A <projectid>" argument.
  For batch jobs add the following line to job scripts:
  #PBS -A <projectid>
• Globus
  In RSL specify "(project=<projectid>)".
  When executing globus-job-run specify the "-p <projectid>" argument.
• Condor-G
  Add the following line to the submit description file:
  globusrsl = (project=<projectid>)
  
  Additional Condor-G documentation