tutorial on AICT cluster

This tutorial shows step by step procedures to run the FCPU code on the AICT cluster at University of Alberta (cluster.srv.ualberta.ca) based on existing petsc-dev built from sjin1 (account of Shi Jin).

Check Read Access

login to cluster.srv.ualberta.ca and make sure you have the read access to the files of sjin1 running the following command:

ls -l /home/sjin1/software/src

The output should include
total 909840
-rw-r--r--   1 sjin1 uofa    183140 May 13  1999 bench.tar.gz
drwxr-xr-x   9 sjin1 uofa      4096 Feb  2  2007 bm
-rw-r--r--   1 sjin1 uofa  35168366 Jun 22  2007 paraview-3.0.2-Linux-x86.tar.gz
drwxr-xr-x  11 sjin1 uofa      4096 Feb  2  2007 petsc-2.3.2-p8
drwxr-xr-x  13 sjin1 uofa      4096 Oct 27 14:41 petsc-dev-g++
-rw-r--r--   1 sjin1 uofa 799969280 Jun 22 11:11 petsc-dev-kalmar.tar
drwxr-xr-x  13 sjin1 uofa      4096 Mar  2  2008 petsc-dev-old
drwxr-xr-x  14 sjin1 uofa      4096 Dec  4 20:27 petsc-dev-opt-g++
drwxr-xr-x  15 sjin1 uofa      4096 Jun 22 11:36 petsc-dev-sieve
-rw-r--r--   1 sjin1 uofa  81582080 Jun 20 23:36 petsc-dev.tar
-rw-r--r--   1 sjin1 uofa  13798421 Oct  2 19:23 petsc-dev.tar.gz
drwxr-xr-x   2 sjin1 uofa      4096 Feb  4  2007 stream

If the line for petsc-dev-opt-g++ is not there, then there is no need to continue this tutorial since you don't have read access to my directory and therefore you should build your own petsc-dev.

Set enviroment variable

Edit your ~/.bashrc file to have the following lines:

module load mpi/openmpi-1.2.5
export PETSC_ARCH=linux-gnu-cxx-opt
export PETSC_DIR=/home/sjin1/software/src/petsc-dev-opt-g++

Once set, log out and login again or run ". ~/.bashrc" to source it.
Note that your default shell on AICT cluster may be csh while the above procedure works for bash. Here is the quote from AICT support
At the moment, permanently changing
your login shell on the cluster is
not possible.

As a workaround, edit your .login
file to contain only one line

  exec bash -i -l

This is what I do.  It seems to work ok.

When submitting jobs to the cluster,
be sure to include the following
directive in your scripts

  #PBS -S /bin/bash

Ed[mund [Sumbar]]
Research Support, Univ of Alberta

Grab the code

To retrieve the latest fcpu source code using subversion, run the following

svn checkout https://svn.xp-dev.com/svn/jinzishuai_fcpu/ new-fcpu-tutorial

The new-fcpu-tutorial is the name of the new directory to be created where the codes are.
Refer to the faq if you need more help on using subversion to get the code.

If you already have the subversion directories checked out, then you should do

svn update

to make sure the code is update-to-date.

Build the code

Run the following commands

cd new-fcpu-tutorial/trunk/

And you will see lots of compilation messages. Once it is done, you should have a file called fcpu under the current directory.

Generate a simple mesh

Build the box mesh generator


make Tet1stOrderBox

in the source directory and you should get a exectuable Tet1stOrderBox generated.

Create input files

Use your favorite text editor (VI, Emacs, etc) to create three files: xs.dis, ys.dis and zs.dis. In this example, their contents are identical and can be be seen as

sjin1@opteron-cluster tutorial $ cat xs.dis
sjin1@opteron-cluster tutorial $

The above is a simple example. You can use a simple Matlab script to generate these kind of meshes automatically.
Its code looks like this
save -ascii xs.dis x;
save -ascii ys.dis y
save -ascii zs.dis z

Generate mesh

Now run the Tet1stOrderBox code in the same directory of the *.dis files. For example

sjin1@opteron-cluster tutorial $ ../Tet1stOrderBox
m = 125 cubes, nm = 216, n = 216 1st-nodes, so there are 750 elements
sucessfully allocated 1736 Bytes for map
node = 216
elem = 750
sjin1@opteron-cluster tutorial $

Here Tet1stOrderBox is located in the parent directory. You may want to use its absolute path.

Now there are a number of files named with 3d.* are generated in the current directory. To see them

sjin1@opteron-cluster tutorial $ ls
3d.aux  3d.bc  3d.lcon  3d.mybc  3d.nodes  genDis.m  xs.dis  ys.dis  zs.dis

Create the particle input file

A sample particle input file is given below

sjin1@opteron-cluster tutorial $ cat 3d.particles
#rho_f=1, t=0
#1-id 2-radius 3-rho_p 4-fixed 5-x 6-y 7-z 8-u 9-v 10-z 11-w1 12-w2 13-w3 14-theta 15-phi
0 0.3 1.2 0 0 0 0.5 0 0 0 0 0 0 0 0
sjin1@opteron-cluster tutorial $

This file is quite self explantory. Now the first three lines are required.

Test run

To see if the fcpu code works, simply run it without any argument like


Note again you might want to use the absolute path of the fcpu file.
My output looks like

If you similar outputs, congratulation! Your code is built successfully.

PBS script

To run a job through the PBS scheduler on the cluster, try the following script

sjin1@opteron-cluster tutorial $ cat submit.pbs
#!/bin/bash -l
#PBS -N test
#PBS -S /bin/bash
#PBS -l nodes=1:ppn=1
#PBS -l walltime=00:010:00
#PBS -l pvmem=2000mb
#PBS -M shi.jin@ualberta.ca

echo "Starting run at: `date`"
mpiexec  -np 1   $CODE -Re 1.0 -Fr 0.0098 -dt 0.05 -NSteps 10 \

echo "Job finished at: `date`"

Once it is run, there will be many outputs in the out1 file. Also many files are genrated.
For example,

  • .pvtu and .vtu files are pressure and velocity fields at a given step. They can be opened by paraview.
  • .msh are GMSH files describing the mesh alone.
  • .pos are GMSH field files to be used together with the .msh file.
  • particles/ directory stores all particle related files. There are two kinds of particle files
    • particle?.dat: describes the trajectory of a given particle
    • parT????.dat: has all particle information at a given time snap shot.
  • restart_*/ directory stores files needed for a restart.


Parallel jobs

As shown in the pbs script, fcpu can be invoked by

mpiexec -n 2 fcpu

Note that for this example to work, we need to change the zs.dis file to only have 0.8 length.
Strictly speaking, the example give above is not a very good example at all since the grid size is comparable to the sphere size and in parallel we require each zone to be larger than the sphere. For a better parallel example, try a better resolved problem, such as the single particle sedimentation experiment done by ten Cate.

However, there are issues with problem size due to memory limitation. Refer to tutorial on parallel simulation for details.

Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License