Cross Compile For Blue Gene L Hdf5 As An Example

Since Blue Gene is a batch system and the front end node (fen) has a different architecture from the compute nodes, cross compiling is required.
This tutorial uses HDF5 as an example to show how cross-compilation is done on Blue Gene.

1. Get HDF5 Source

cd /gpfs/bglscratch/pi/sjin/src
wget ftp://ftp.hdfgroup.org/HDF5/current/src/hdf5-1.8.2.tar.gz
tar -xzf hdf5-1.8.2.tar.gz

Note that the home directory is not visible from the compute nodes, everything has to be located inside /gpfs/bglscratch/pi/, which is shared to all nodes.

2. Change the configuration script

cd /gpfs/bglscratch/pi/sjin/src/
mv hdf5-1.8.2 hdf5-1.8.2_cross
cp configure configure.orig
./bin/yodconfigure configure

What yodconfigure does is to simply prefix all serial executions with the
contents of the RUNSERIAL variable. It just rewrites part of the configure
script, so you still have to run configure after using yodconfigure. So,
for example, instead of executing:

./a.out

after being run through yodconfigure, the configure script will execute:

RUNSERIAL ./a.out

And if runserial is empty, it's equivalent to the original configure, BUT if
RUNSERIAL is set to something like mpirun -np 1 the command would become:

mpirun -partition R001 -np 1 -cwd $PWD -exe a.out

which ensures that the binary is run on a compute node rather than the login
node.

To make lives easier, a script called run1.sh is created to run a serial code. Its content is

#!/usr/bin/env bash
#
# Copyright 2008 Adam C. Lichtl
# RIKEN BNL Research Center
# Brookhaven National Laboratory
#
# run1.sh
#
# Note: this script can not be called interactively on an IBM Blue Gene/L,
#       it must be invoked from inside of a LoadLeveler job
# Note: changed by Shi Jin so that it works interactively without LoadLever by put the partition in.
# Note: The partition used here is R001-N1-32, and it is not de-allocated after the mpirun. So after the run, the user would manualy free the partition by calling:
#ssh sn freepartition -partition R001-N1-32

if [ $# -lt 1 ]; then
    echo "Usage: $0 <executable> [<arguments>]" >&2
    exit 1
fi

cwd=`pwd`
if [ $# -eq 1 ]; then
  mpirun  -partition R001-N1-32 -np 1 -cwd $cwd -exe $1 -nofree
else
  exe=$1
  shift
  mpirun -partition R001-N1-32 -np 1 -cwd $cwd -exe $exe -args "${*}" -nofree
fi

#ssh sn freepartition -partition R001-N1-32

exit 0

Then to run a serial code (say a.out) on an arbitrary compute node, simply do

run1.sh a.out

assuming run1.sh is in the path.

3. Configure HDF5

Run the modified script

export RUNSERIAL=run1.sh
./configure --disable-stream-vfd CC=mpixlc

Since the test codes (conftest) are now running on the compute nodes, it takes a long time to finish the configure steps.
More options for ./configure can be supplied to suit specific needs. Typical ones are —prefix —enable-parallel —enable-fortran , and etc.

4. Build

make
make install

References

Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License