Since Blue Gene is a batch system and the front end node (fen) has a different architecture from the compute nodes, cross compiling is required.
This tutorial uses HDF5 as an example to show how cross-compilation is done on Blue Gene.
1. Get HDF5 Source
cd /gpfs/bglscratch/pi/sjin/src
wget ftp://ftp.hdfgroup.org/HDF5/current/src/hdf5-1.8.2.tar.gz
tar -xzf hdf5-1.8.2.tar.gz
Note that the home directory is not visible from the compute nodes, everything has to be located inside /gpfs/bglscratch/pi/, which is shared to all nodes.
2. Change the configuration script
cd /gpfs/bglscratch/pi/sjin/src/
mv hdf5-1.8.2 hdf5-1.8.2_cross
cp configure configure.orig
./bin/yodconfigure configure
What yodconfigure does is to simply prefix all serial executions with the
contents of the RUNSERIAL variable. It just rewrites part of the configure
script, so you still have to run configure after using yodconfigure. So,
for example, instead of executing:
./a.out
after being run through yodconfigure, the configure script will execute:
RUNSERIAL ./a.out
And if runserial is empty, it's equivalent to the original configure, BUT if
RUNSERIAL is set to something like mpirun -np 1 the command would become:
mpirun -partition R001 -np 1 -cwd $PWD -exe a.out
which ensures that the binary is run on a compute node rather than the login
node.
To make lives easier, a script called run1.sh is created to run a serial code. Its content is
#!/usr/bin/env bash
#
# Copyright 2008 Adam C. Lichtl
# RIKEN BNL Research Center
# Brookhaven National Laboratory
#
# run1.sh
#
# Note: this script can not be called interactively on an IBM Blue Gene/L,
# it must be invoked from inside of a LoadLeveler job
# Note: changed by Shi Jin so that it works interactively without LoadLever by put the partition in.
# Note: The partition used here is R001-N1-32, and it is not de-allocated after the mpirun. So after the run, the user would manualy free the partition by calling:
#ssh sn freepartition -partition R001-N1-32
if [ $# -lt 1 ]; then
echo "Usage: $0 <executable> [<arguments>]" >&2
exit 1
fi
cwd=`pwd`
if [ $# -eq 1 ]; then
mpirun -partition R001-N1-32 -np 1 -cwd $cwd -exe $1 -nofree
else
exe=$1
shift
mpirun -partition R001-N1-32 -np 1 -cwd $cwd -exe $exe -args "${*}" -nofree
fi
#ssh sn freepartition -partition R001-N1-32
exit 0
Then to run a serial code (say a.out) on an arbitrary compute node, simply do
run1.sh a.out
assuming run1.sh is in the path.
3. Configure HDF5
Run the modified script
export RUNSERIAL=run1.sh
./configure --disable-stream-vfd CC=mpixlc
Since the test codes (conftest) are now running on the compute nodes, it takes a long time to finish the configure steps.
More options for ./configure can be supplied to suit specific needs. Typical ones are —prefix —enable-parallel —enable-fortran , and etc.
4. Build
make
make install