Quick User Guide

Cluster Access

•  To gain access to the cluster, please contact Viktor Oudovenko (Physics, Room E256) and provide the following information:
• Your full name
• Username (should match your Rutgers NetID)
• Hostname or IP address of the computer you will use to log in
• Room number and phone number
• Expected period of cluster use
• Name of the professor you work with
Please read the information on this web page carefully and be prepared to answer three questions about the cluster usage policy before receiving a password for your account.

• Log in to the cluster through the front-end node (rupc-01.rutgers.edu)
• Access to the cluster is allowed   only from authorized computers and computers connected through the Rutgers VPN. 
• VPN configuration: Follow the OpenVPN client setup guide to configure your computer. Ask the cluster administrator for your .ovpn profile. For related documentation, see VPN documentation. The OpenVPN profile described here is supplied by the cluster administrator; it is separate from the Rutgers university VPN service.
• Secure Shell (SSH) is required to access the cluster and to log in to machines within it.
• To configure passwordless access within the cluster, run the provided script line by line.
• Secure Copy (SCP) is required to transfer data to, from, and within the cluster.  
• Interactive jobs are not permitted on login nodes.
• Each user may have no more than seven open login sessions on a front-end machine.

Cluster Hardware

• Main Servers 
• Main file server [home directories]
• LDAP server
• Work file servers [directories for large working datasets]
• Backup Servers
• SGE servers [queuing system]
  
• Web and temperature control servers

Cluster Software

• Operating Systems
  Rocky Linux 9.3

• Modules
  The module command provides access to the Modules package, which uses modulefiles to modify your environment dynamically.

• module available                       — list the modules available on the system.
• module load   module_name — load a module.
• module rm     module_name — unload a module.
• module show   module_name — display information about a module.

• Compilers
  C, C++, Fortran 77, and Fortran 90 compilers, the OpenMPI parallel environment, and numerical libraries are available throughout the cluster.

•  Intel oneAPI 2024.0 
  
•  GNU
  

• Libraries
  Commonly used libraries include MKL, FFTW, HDF5, and GSL.  

• Jupyter Notebook

  To access Jupyter Notebook, follow the setup instructions on this webpage.

• Makefile Examples for Frequently Used Software

  • ABINIT
  • FFTW
  • DMFT
  • Espresso
  • VASP
  • Wannier90
  • WIEN2K

Queuing System

To run serial or parallel jobs on the cluster, prepare a job control file and submit it from the submission host, rupc-01. A job control file is a shell script with additional directives specifying the queue, number of CPUs, and other resources. We recommend using the job script templates below.

Queues active as of September 1, 2024:

• CMT (submission host: rupc-01)

        
CLUSTER QUEUE                   CQLOAD   USED    RES  AVAIL  TOTAL aoACDS  cdsuE  
--------------------------------------------------------------------------------
all.q                             0.11      0      0      0   5968      0   5968 
ari56m4                           0.02     24      0    312    336      0      0 
dki28m4                           0.00      0      0    140    140      0      0 
dki28m4d                          0.00      0      0    112    112      0      0 
dki36m5                           0.00      0      0     72     72      0      0 
dko48m4                           0.00      0      0    768    864      0     96 
dko64m4                           0.00      0      0    192    192      0      0 
gkbnl                             0.35      0      0    612    648      0     36 
gkbnlg                            0.00      0      0    144    144      0      0 
gki16m4                           0.00      0      0    256    256      0      0 
gki16m6                           0.00      0      0    432    528      0     96 
gki24m5                           0.00      0      0    648    696      0     48 
gki28m4                           0.00      0      0    112    112      0      0 
gki36m5                           0.49    579      0    249    828      0      0 
gko64m4                           0.00      0      0    640    640      0      0 
jpi36m5                           0.00      0      0    252    252      0      0 
jpo64m4                           0.00      0      0    384    384      0      0 
        
        
        

• AR — Ananda Roy
• DK — David (Vanderbilt) and Karin (Rabe)
• GK — Gabriel (Kotliar) and Kristjan (Haule)
• JP — Jedediah Pixley

• Job Control Script Templates

  • submit_ompi.sh   — a submission script for jobs in all queues.
  • For serial jobs in any queue, use submit.sh .

  
  Guidelines for editing a job control script:

  1.  Replace JOB_NAME with a meaningful job name.
  2.  NUMBER_OF_CPUS in the #$ -pe line can be a single number (e.g., 8) or a range (e.g., 4-32).
  3. Replace the email address YOUR_E_MAIL@physics.rutgers.edu with your own address to receive notifications when the job starts or ends.
  4. Finally, replace ./YOUR_EXEC with the path to your executable.

• Basic Job Management Commands

  • To submit a job:

  qsub  your_job_script.sh 

  •  To check the status of a submitted job:

    qstat

(qw = queued, t = starting, r = running)

•  To delete a job from a queue:

  qdel your_job_number 

•  To view active queues and their load:

  qstat -g c 

  (alias qs)

•  To check whether enough resources are available to run your job:

  qsub -w v job_script.sh

Job output:
Job files are located in /scr on the compute node. While a job is running, its terminal output is written to /tmp. When the job finishes, that output is moved to /scr. To view files while a job is running, log in to the node running it. The job directory is named Job_numberUsername (for example, 12345username).

• Tips and Tricks

• To submit a job to a particular host:

  qsub -l hostname=n17 job_script.sh

• To submit a job with a time limit (one hour in this example):

qsub -l s_cpu=01:00:00 job_script.sh

• To submit a job to nodes with 2 GB of memory:  

  qsub -l mem_total=2000M job_script.sh

• To submit a job at a specific time:  

  qsub -a yyyymmddhhmmss job_script.sh

  (For example, -a 20041230153234 specifies December 30, 2004, at 15:32:34.)

• The graphical utility qmon (Queue Monitor) lets you submit jobs and view information about the cluster. 
• For more information about SGE, see the User Guide and Administrator Guide as well as the presentation and online resources.
  
• To access a compute node in the CoRE cluster, use the command: ssh  nX .
  

Backups

Home and work directories are backed up once a day.

Storage Policy

Cluster Usage Policy

• All new cluster users must answer at least three questions about the cluster usage policy before being granted access.
  
• Before leaving Rutgers, notify the cluster administrator of your departure date. If you do not, your cluster access will be blocked as soon as the administrator learns of your departure through other channels.
  
• After the departure date, users have three months to archive their data and move it from the active work area to the storage directory.
• Accounts will be closed six months after the user's departure date unless their supervisor confirms that they still have active projects.
  

• Users who exceed their quota receive three automatic reminders to reduce disk usage. After the third reminder, their account will be suspended.
• Store data as tar archives. Both uncompressed .tar files and gzip-compressed .tar.gz or .tgz files are permitted in the storage directory.