Stern Center for Research Computing

Introduction

SCRC supports a moderate size collection of computers, called the Stern GRID, which allow researchers to attack problems that
are computationally challenging or that require large amounts of disk storage. To distribute the software and hardware resources equitably a scheduling system called the Sun Grid Engine (SGE) is used. For statistics on usage see Stern GRID usage reports.

How Do I Know If I Should Use the Stern GRID

Here are some typical examples for using the Stern GRID.

• You expect your job to run longer than an hour.
• You expect your job needs a large amount of disk space or processing power.
• You have many jobs and would like them to run on several computers concurrently to speed up processing time.
• You want to run the same program with many different datasets.
• You want to run the same program but need to systematically change a parameter inside the program.
• You need to use a particular application that is installed only on specific computers in the Stern GRID.

Sun Grid Engine (SGE)

The Sun Grid Engine (SGE) is a sophisticated scheduling system for distributing jobs across a heterogeneous group (a grid) of cooperating computers. Users of the Stern GRID need only know about the requirements of their job, and need not know about all of the different computers that might be able to process their job.
Users describe the processing they need to do by creating a small SGE submission script that tells the SGE software how to run the job and what the special characteristics of the job are. They then submit their request via the shell script to the SGE. The SGE finds the computer(s) that best satisfy the user’s requirements and runs the job.

Resource Distribution

This approach allows efficient use of all the resources available, as well as providing the ability to make sure that resources are used equitably (i.e. one user can’t take over all of the resources). This is the standard approach used in almost all high performance computing environments.

Array Task Capabilities

The SGE is particularly effective for array tasks, such as simulation, that can be divided into a number of smaller independent pieces. Each piece can run simultaneously on separate machines.

For example, suppose you want to run a simulation with 120,000 trials, and each trial will take 6 seconds of processing. In a normal environment, your simulation will take 720,000 seconds, or 12,000 minutes, or 200 hours of processing. This would take weeks to run. However, if you divide your simulation into 10 runs, each of which does 12,000 trials, you will have your output in 1/10th of the time, i.e. days instead of weeks. See “Array job” below for an example.
Another type of array task is a sequenced array job, where, the processing order of the runs can be specified. For example, say you have a post-processing run that uses as input the output of 10 other runs, i.e., all 10 runs must complete before the 11th post-processing run can start. The SGE can be configured so that you submit all 10+1 runs together, it knows to wait until the first 10 runs are finished before executing the 11th post-processing run. See Sequenced Array Job for an example.

Getting started

Logging In

To log in to the Stern GRID see Stern GRID Access. For Linux basics see Linux Basics.

Your Home Directory

Once logged in to the Stern GRID, you are by default in your home directory. Your home directory resides on a server that is separate to but accessible from any computer in the Stern GRID.

Using the SGE

To use the SGE, you must create a “submission script” that contains the commands and programs you want to execute. The script may also contain options or instructions that control the behavior of the SGE.

Simple Job

The most basic type of SGE submission script has no options. It consists entirely of one or more commands to execute. While you will likely never use such a simple script, it illustrates the SGE qsub concept. For example, the following script executes one command and prints “hello world” to the standard output. Make sure that this script is executable (chmod a+rx hello.sge); see Linux Basics for an explanation about file permissions. Try it.

This script does not set any SGE options. You submit your script (or job) to the SGE queuing system using the qsub command.

After submission, you should see a message similar to:

Once the job has completed, two files are created and placed in your home directory:

where # is the job ID assigned by SGE. The e# file contains output from standard error and the o# file contains output from standard out. If your job ran successfully, the e# file should be empty and the o# file should contain the text “hello world”.

Job With Options

Options define the specific requirements of the job to the SGE. A typical job will have options. In the submission script, option lines begin with #$. Try it.

You submit the job as before with the command:

You can see the status of your job by typing the qstat command. If the job is running, qstat reports something similar to this:

Here you can see that the job called “helloJob” has a unique job ID of 55397, and the state of the job is r(unning).

Array Job

Array jobs are a set of related jobs. Every job in the collection is assigned a task ID which can be used in the job script to control the behavior of the job. This method works well for simulation tasks where you want to repeat a job many times each with a different input file.
For jobs running SAS, R, and matlab, the task ID can be used inside the program to control the behavior of the program. See examples for SAS, R, and Matlab.
Consider the following submission script called array.sge. We want to submit the C program called analysis to the SGE with 100 different data sets (data.1 to data.100).

You submit the job using qsub and the parameter -t to specify the range of task IDs.

The SGE will run the job 100 times, varying the environment variable $SGE_TASK_ID from 1 to 100. The SGE system schedules concurrently as many jobs as it can depending on the current availability of computers in the Stern GRID. Each job has its own o# and e# file in the form:

where # is the job ID and the $ is the task ID, i.e., analysisJob.e(job_ID).(task_ID).

Sequenced Array Job

Running jobs sequentially rather than concurrently is very useful for multistage processing. For example say we run 10 jobs concurrently using the array method. Next, we want to post-process the output of the first 10 jobs into a single result. We can specify that the post-processing job not begin until the first 10 jobs have completed.

To have a job wait for another job, add the -hold_jid job1_ID option to your post-processing script.

As an example, say we have two scripts sequence1.sge and sequence2.sge. The first script, sequence1.sge runs 10 times. Each task takes a random length of time to complete. It then appends a random number to result.list. The second script, sequence2.sge runs after all of the sequence1.sge tasks have completed. It sorts result.list into result.sorted. Try it.

Now, submit both jobs to the SGE.

Typing qstat directly after submitting the jobs shows that they are q(ueued) and w(aiting) to be scheduled. Additionally stage2Job has a h(old) status.

After a few seconds, qstat shows most of the tasks running for stage1Job while stage2Job is still on hold.

A peek into the output files for each stage shows that stage 2 did not start until all tasks from stage 1 finished.

Running a Matlab Job

In this example we use Matlab’s Financial Toolbox to compute the Black-Scholes put and call option pricing. This example also shows how matlab uses the SGE Task ID variable in the program to vary a parameter value. Try it.

Submit the job using qsub and the parameter -t to specify the range of task IDs; increment them by 2, i.e., 30, 32, 34, etc.

The Matlab program

Ouput for SGE_TASK_ID = 30 which converts to volatility = 0.3

call = 10.1592
put = 2.8136

Running Parallel Matlab Jobs

Many of the new software applications will automatically use multiple cores (processors) if they are available. This works fine in a dedicated system environment, but plays havoc with the scheduling of jobs on the GRID. It is easy to completely overload one of our nodes by scheduling 8 4 core jobs on a node that only has 8 processors. This has been happening frequently with the more recent versions of matlab.

But there is an easy solution. Tell the GRID scheduler how many cores your job will need, and then restrict matlab to that many cores. In the simplest case, assume that your job usually uses about 4 cores (you can see this by running top while your job is running), then just reserve 4 cores and place your job in a parallel queue.

You can do this by adding two lines to your job script.

This will place your job in the P4 queue, which allows up to 4 cores per job.
The PE4 4 tells Sun Grid Engine that you need 4 cores for this job.

You could also do

and you will get between 2 and 4 cores.

In matlab, you can fix the number of cores you use to what was available by issuing the following commands

Note that the PE4 parallel environment will limit you to a max of 4 cores. If you really need more, contact the staff at research@stern.nyu.edu.

Java also tends to use multiple threads, so you should use the same technique with large java jobs.

Running a SAS Job

In this example we use data about the right- or left-handedness observed in men and women and create a cross tabulation of gender versus handedness.Try it.

Submit the job using qsub.

The SAS program

Here is the cross tabulation output found in the file crosstab.lst.

Running a Stata Job

Here is the SGE submit script for running a simple Stata job. Try it.

The data file, “testData.raw”

The data dictionary specifying format of fixed data file, “testData.dct”

The Stata program.

Running an R Job

In this example we fit a smooth spline model. Try it.

Submit the job using qsub and the parameter -t to specify the range of task IDs; i.e., 5, 10, 15, 20, 15

The R program

The output is a a set of graphs in postscript files. For example, when
the task ID = 10, the error variance is v = 10/100 = 0.1 and generates
the following random model data and the program fits a smooth spline to this data.
This graph is from the result_10.ps file.

More SGE Topics

Monitoring a job with qstat

The qstat command shows the status of your job(s) as well as the available queues. Here are some options to qstat.

Example 1

Here, qstat shows that the job named fitsplineJob with job ID 56655 has three state indicators – E(rror), q(ueued), and w(aiting). It also shows that the it is a multi-task job with task numbers incrementing by 10, i.e., 10, 20, .., 60.

Example 2

Here we see fitsplineJob is now running the tasks 10, 20, …, 60 simultaneously on six different queues.

Job States

Queue states

Passing an environment variable to a job

You can pass user defined environment variables to a program by using the -v argument.

Retrieve the variable in your program with getenv. See Running a Matlab Job and Running an R Job for examples.

Passing entire environment to a job

Retrieve any and all environment variables defined at qsub time. See the Linux set command for a complete list.

Summary of Commands, Options and Resources

Commands

Options

Resources (see “qconf -sc” for a complete list)

Troubleshooting