Using MPICH on the Janus Cluster

Overview

• Each node has 12 cores (2 cpus) (intel eon 2.8 Ghz)
• 2 GB of RAM per core, 24 GB of RAM per node
• QDR infiniband interconnect

 File System

/home/<username> (2 GB quota)

/projects/<username> (256 GB quota, not for job output)
/scratch/stmp00/<username> (not backed up, intended for job I/O)

Steps to run an MPI Program
1) Load Dotkit for MPI
use .openmpi-1.4.3_ib

2) compile

mpicc -std=c99 -o <outfile> <inputfile>

The -sdd=c99 is needed if you want to use C99 (you probably do).  With c99, you can do a loop like:
for(int i = 0; i <n; i++) {...}
Without c99, you have to write this loop as:
int i;
for(i = 0; i < n; i++) {...}

3a) Submit

use Torque

use Moab

3b) write a batch script to run the mpi program (must be in PBS standard) parallel batch scheduler?

3c) submit:

qsub -q janus-debug run.sh

-q = run queue

rc.colorado.edu/crcdocs/queues

4) track status
qstat -f -u $USER

showq -u $USER

5) Kill job

qdel [jobid]

mjobctl -c [jobid]

Using MPICH on Windows 7 with MS Visual Studio 2010

Installing MPICH2

• Download 32-bit or 64-bit MPICH installer (if you have Visual Studio 32-bit, you MUST use the 32-bit MPICH) from http://www.mcs.anl.gov/research/projects/mpich2/downloads/index.php?s=downloads
• Open the start menu
• Type "cmd"; this will bring up cmd.exe in the search results
• Right-click on the cmd.exe icon and click "Run as admistrator"
• Change directories to the location of the .msi file you just downloaded
• Execute the .msi file by entering it's name on the command prompt.
• You may be prompted with a security messgae from windows, continue
• Make sure to remember the "authentication passphrase for smpd".  You must have this to run a program with mpich.

NOTE: If you have problems with installing MPICH2 or running it, you may need to lower the "User Account Control" settings on Windows 7.  You can do so by doing the following:

1.  Open the Start Menu
2. Type "User Account Control"
3. Choose "Change User Account Control Settings" from the search results.
4. You can lower the slider to the bottom to reduce the security level of Windows 7.
5. Install MPICH
6. Reset the security level back to what it was before.

Setting up Execution Service
After installing MPICH, check that the smpd.exe service is installed properly (this DOES NOT happen by default): check to see if there is a windows service called "MPICH2 Process Manager, Argonne National Lab".  If it is not, do the following to install it (assumes 32-bit, remove (x86) for 64-bit):

• Open a command prompt
• Change directories to C:\Program Files (x86)\MPICH2\bin
• run "smpd -install"; the output will be: "MPICH2 Process Manager, Argonne National Lab installed."

Setting up Visual Studio
First, make sure that your Visual Studio version and MPICH version are using the same architecture.  That is, if you have VS 2010 32-bit (the default student edition), you MUST have the 32-bit MPICH installed.  You can check this by the MPICH installation directory.  If MPICH is installed in C:\Program Files (x86), then you're using the 32-bit version; whereas, if it's installed in C:\Program Files, you're using the 64-bit version.

Once you're ready to code, setup the Visual Studio Project to use MPICH:

1. Right click on the solution in the "Solution Explorer" window; choose properties (figure 1).
2. Under "Configuration Properties", choose "VC++ Directories".
3. Highlight the "Include Directories" row at the right.
4. Click the down arrow at the far right of the highlighted row.
5. Click "Edit<...>"
6. Click the "New Line" icon (a folder with a start on it) at the top of the "Include Directories" dialog box.
7. Click the "..." button.
8. Add the MPICH include file location; for a 32-bit build, this is C:\Program Files (x86)\MPICH2\include, for a 64-bit build, this is C:\Program Files\MPICH2\include
9. Click "Select Folder".  Click "OK".
10. Highlight the "Library Directories" row.
11. Repeat steps 4 through 9 for the library directory of MPICH: C:\Program Files (x86)\MPICH2\lib or C:\Program Files\MPICH2\lib
12. Under "C/C++", choose "General".
13. Highlight "Additional Include Directories".
14. Repeat steps 4 through 9 for the include directory of MPICH: C:\Program Files (x86)\MPICH2\include or C:\Program Files\MPICH2\include
15. Under "Linker", choose "General".
16. Highlight "Additional Library Directories".
17. Repeat steps 4 through 9 for the libary directory of MPICH.
18. Under "Input", choose "Input".
19. Highlight "Additional Dependencies".
20. Click the down arrow at the far right of the highlighted row.
21. Click "Edit<...>"
22. Add "cxx.lib" and "mpi.lib" (don't include the quotes).
23. Click "OK".
24. Click "OK" in the " Property Pages" window.

Figure 1: The Project Properties

Setting up the MPICH execution Environment

1. Open a command prompt
2. Execute "mpiexec -register" (enter your windows password for the password)
3. Validate the user by running "mpiexec -validate -user <username>"

Running an MPI program with MPICH

• Add the MPICH bin directory, C:\Program Files (x86)\MPICH2\bin, to your path
• Find the .exe file created by VS 2010.  I had to manually set the location of the .exe file in the "Output" section's "General" tab in the properties of the solution.
• If you have trouble, use the cpi.exe example (this is the first code example in the Using MPI book):

NOTE: I was unable to get more than 1 process running.  I've emailed the MPICH discussion list to see if anybody knows what's wrong.