Introduction
Login
With SSH key only. Need to setup the key pair first.
Compiler Version
Ref: http://www.cisl.ucar.edu/docs/frost/cbr.jsp#compiling
Note:
mpxlc=mpixlc
guntaka@fr0103en:~/src/petsc-3.1-p0> mpicxx -show
/bgl/BlueLight/V1R3M4_300_2008-080728/ppc/blrts-gnu/bin/powerpc-bgl-blrts-gnu-g++ -I/bgl/BlueLight/V1R3M4_300_2008-080728/ppc/bglsys/include -L/bgl/BlueLight/V1R3M4_300_2008-080728/ppc/bglsys/lib -lcxxmpich.rts -lmpich.rts -L/bgl/BlueLight/V1R3M4_300_2008-080728/ppc/bglsys/lib -lmsglayer.rts -ldevices.rts -lrts.rts -ldevices.rts -lrts.rts
guntaka@fr0103en:~/src/petsc-3.1-p0> /bgl/BlueLight/V1R3M4_300_2008-080728/ppc/blrts-gnu/bin/powerpc-bgl-blrts-gnu-g++ --version
powerpc-bgl-blrts-gnu-g++ (GCC) 3.4.3
Copyright (C) 2004 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Cobalt Scheduler
Ref: http://www.cisl.ucar.edu/docs/frost/cbr.jsp#Submission
- cqstat -q: see a list of queues
guntaka@fr0103en:~/src/petsc-3.1-p0> cqstat -q
Name State Users MinTime MaxTime MaxRunning MaxQueued MaxUserNodes TotalNodes Priority
=========================================================================================================================================================
default running None None 24:00:00 12 20 None None 0
debug running None None 02:00:00 8 20 None None 0
- cqsub -n <power of 2> -t <h:m:s> <program>
PETSc
C version without Sieve
The simplest case:
- ./configure —with-batch —download-f-blas-lapack=1
- cqsub -q debug -n 1 -t 00:10:00 conftest
- ./reconfigure.py
output
Compilers:
C Compiler: mpicc -Wall -Wwrite-strings -Wno-strict-aliasing -g3
Fortran Compiler: mpif77 -Wall -Wno-unused-variable -g
Linkers:
Static linker: /usr/bin/ar cr
MPI:
Includes: -I/bgl/BlueLight/V1R3M4_300_2008-080728/ppc/bglsys/include
BLAS/LAPACK: -Wl,-rpath,/home/guntaka/src/petsc-3.1-p0/linux-gnu-c-debug/lib -L/home/guntaka/src/petsc-3.1-p0/linux-gnu-c-debug/lib -lflapack -Wl,-rpath,/home/guntaka/src/petsc-3.1-p0/linux-gnu-c-debug/lib -L/home/guntaka/src/petsc-3.1-p0/linux-gnu-c-debug/lib -lfblas
PETSc:
PETSC_ARCH: linux-gnu-c-debug
PETSC_DIR: /home/guntaka/src/petsc-3.1-p0
Clanguage: C
Scalar type: real
Precision: double
shared libraries: disabled
dynamic libraries: disabled
Memory alignment: 16
xxx=========================================================================xxx
Configure stage complete. Now build PETSc libraries with:
make PETSC_DIR=/home/guntaka/src/petsc-3.1-p0 PETSC_ARCH=linux-gnu-c-debug all
xxx=========================================================================xxx
One could also get it to work with mpxlc
C++ Version with Sieve
- ./configure —with-clanguage=C++ —with-sieve=1 —with-opt-sieve=1 —download-chaco=1 —download-boost=1 —with-shared=0 —with-debugging=no —with-mpi-shared=0 —download-f-blas-lapack=1 —with-batch
- cqsub -q debug -n 1 -t 00:10:00 conftest. Make sure the job is finished by "cqstat".
- ./reconfigure.py
Output:
Compilers:
C Compiler: mpicc -Wall -Wwrite-strings -Wno-strict-aliasing -O
C++ Compiler: mpicxx -Wall -Wwrite-strings -Wno-strict-aliasing -O
Fortran Compiler: mpif77 -Wall -Wno-unused-variable -O
Linkers:
Static linker: /usr/bin/ar cr
MPI:
Includes: -I/bgl/BlueLight/V1R3M4_300_2008-080728/ppc/bglsys/include
BLAS/LAPACK: -Wl,-rpath,/home/guntaka/src/petsc-3.1-p0/linux-gnu-cxx-opt/lib -L/home/guntaka/src/petsc-3.1-p0/linux-gnu-cxx-opt/lib -lflapack -Wl,-rpath,/home/guntaka/src/petsc-3.1-p0/linux-gnu-cxx-opt/lib -L/home/guntaka/src/petsc-3.1-p0/linux-gnu-cxx-opt/lib -lfblas
Chaco:
Library: -Wl,-rpath,/home/guntaka/src/petsc-3.1-p0/linux-gnu-cxx-opt/lib -L/home/guntaka/src/petsc-3.1-p0/linux-gnu-cxx-opt/lib -lchaco
Sieve:
Includes: -I/home/guntaka/src/petsc-3.1-p0/include/sieve
Boost:
Includes: -I/home/guntaka/src/petsc-3.1-p0/externalpackages/Boost/ -I/bgl/BlueLight/V1R3M4_300_2008-080728/ppc/bglsys/include
PETSc:
PETSC_ARCH: linux-gnu-cxx-opt
PETSC_DIR: /home/guntaka/src/petsc-3.1-p0
Clanguage: Cxx
Scalar type: real
Precision: double
shared libraries: disabled
dynamic libraries: disabled
Memory alignment: 16
xxx=========================================================================xxx
Configure stage complete. Now build PETSc libraries with:
make PETSC_DIR=/home/guntaka/src/petsc-3.1-p0 PETSC_ARCH=linux-gnu-cxx-opt all
xxx=========================================================================xxx
make error
/home/guntaka/src/petsc-3.1-p0/include/sieve/ALE_args.hh: In member function `ALE::ArgValue ALE::ArgDB::operator[](const string&) const':
/home/guntaka/src/petsc-3.1-p0/include/sieve/ALE_args.hh:132: error: call of overloaded `ArgValue(ALE::ArgValue)' is ambiguous
/home/guntaka/src/petsc-3.1-p0/include/sieve/ALE_args.hh:64: note: candidates are: ALE::ArgValue::ArgValue(const ALE::ArgValue&)
/home/guntaka/src/petsc-3.1-p0/include/sieve/ALE_args.hh:67: note: ALE::ArgValue::ArgValue(const boost::program_options::variable_value&)
/home/guntaka/src/petsc-3.1-p0/include/sieve/ALE_args.hh:132: error: initializing temporary from result of `ALE::ArgValue::ArgValue(const boost::program_options::variable_value&)'
From Matt:
this is a bug in the IBM compiler version I have had other places. In the latest
petsc-dev, I just commented out the #include of ALE_args.hh in ALE.hh. This fixes
it. Everyone hates compiler bugs.
Solution: use petsc-dev.
petsc-dev with sieve
mercurial
- wget http://mercurial.selenic.com/release/mercurial-1.5.tar.gz
- vi src/mercurial-1.5/Makefile
PYTHON=python2.5
- vi ~/.bashrc
alias python=python2.5
export PYTHONPATH=/home/guntaka/lib/python2.5/site-packages
export PATH=$PATH:$HOME/bin
- hg clone http://petsc.cs.iit.edu/petsc/petsc-dev
- cd petsc-dev/config
- hg clone http://petsc.cs.iit.edu/petsc/BuildSystem BuildSystem
petsc-dev
The same as above.
vi ~/.bashrc
export PETSC_DIR=/home/guntaka/src/petsc-dev
export PETSC_ARCH=linux-gnu-cxx-opt
fcpu
All the .cxx codes are compiled successfully into objects. But there is an error at the final linking stage:
mpicxx -Wall -Wwrite-strings -Wno-strict-aliasing -O -o fcpu main.o Options.o USG.o Wall.o PlaneWall.o BC.o Quadrature.o Tetrahedron.o TetrahedronTaylorHood.o Cylinder.o Sphere.o CylinderWall.o CFD.o NSSolver.o ParticulateFlow.o BodyFitMeshPF.o CollisionBox.o GranularFlow.o -Wl,-rpath,/home/guntaka/src/petsc-dev/linux-gnu-cxx-opt/lib -L/home/guntaka/src/petsc-dev/linux-gnu-cxx-opt/lib -lpetsc -Wl,-rpath,/home/guntaka/src/petsc-dev/linux-gnu-cxx-opt/lib -L/home/guntaka/src/petsc-dev/linux-gnu-cxx-opt/lib -lchaco -lflapack -lfblas -lm -L/bgl/BlueLight/V1R3M4_300_2008-080728/ppc/bglsys/lib -L/bgl/BlueLight/V1R3M4_300_2008-080728/ppc/blrts-gnu/lib/gcc/powerpc-bgl-blrts-gnu/3.4.3 -L/bgl/BlueLight/V1R3M4_300_2008-080728/ppc/blrts-gnu/powerpc-bgl-blrts-gnu/lib -lmpich.rts -lmsglayer.rts -ldevices.rts -lrts.rts -lg2c -lm -lm -lcxxmpich.rts -lstdc++ -lmpich.rts -lmsglayer.rts -ldevices.rts -lrts.rts -Wl,-rpath,/home/guntaka/src/petsc-dev/linux-gnu-cxx-opt/lib -L/home/guntaka/src/petsc-dev/linux-gnu-cxx-opt/lib -lpetsc -Wl,-rpath,/home/guntaka/src/petsc-dev/linux-gnu-cxx-opt/lib -L/home/guntaka/src/petsc-dev/linux-gnu-cxx-opt/lib -lchaco -lflapack -lfblas -lm -L/bgl/BlueLight/V1R3M4_300_2008-080728/ppc/bglsys/lib -L/bgl/BlueLight/V1R3M4_300_2008-080728/ppc/blrts-gnu/lib/gcc/powerpc-bgl-blrts-gnu/3.4.3 -L/bgl/BlueLight/V1R3M4_300_2008-080728/ppc/blrts-gnu/powerpc-bgl-blrts-gnu/lib -lmpich.rts -lmsglayer.rts -ldevices.rts -lrts.rts -lg2c -lm -lm -lcxxmpich.rts -lstdc++ -lmpich.rts -lmsglayer.rts -ldevices.rts -lrts.rts
collect2: ld terminated with signal 11 [Segmentation fault]
make: *** [fcpu] Error 1
petsc-dev-sieve with IBM compilers
./configure —with-clanguage=C++ —with-sieve=1 —with-opt-sieve=1 —download-chaco=1 —download-boost=1 —with-shared=0 —with-debugging=no —with-mpi-shared=0 —download-f-blas-lapack=1 —with-batch —with-cc=mpxlc —with-cxx=mpxlC —with-fc=mpxlf
Note: —with-fc=mpxlf90 does not work (in building blas-lapack).
cqsub -q debug -n 1 -t 00:10:00 conftest
./reconfigure.py
output:
Compilers:
C Compiler: mpxlc -O
C++ Compiler: mpxlC -qrtti=dyna -O -+
Fortran Compiler: mpxlf -O
Linkers:
Static linker: /usr/bin/ar cr
MPI:
Library: -lnsl -lrt
Chaco:
Library: -Wl,-rpath,/home/guntaka/src/petsc-dev-xlc/linux-gnu-cxx-opt/lib -L/home/guntaka/src/petsc-dev-xlc/linux-gnu-cxx-opt/lib -lchaco
BLAS/LAPACK: -Wl,-rpath,/home/guntaka/src/petsc-dev-xlc/linux-gnu-cxx-opt/lib -L/home/guntaka/src/petsc-dev-xlc/linux-gnu-cxx-opt/lib -lflapack -Wl,-rpath,/home/guntaka/src/petsc-dev-xlc/linux-gnu-cxx-opt/lib -L/home/guntaka/src/petsc-dev-xlc/linux-gnu-cxx-opt/lib -lfblas
Sieve:
Includes: -I/home/guntaka/src/petsc-dev-xlc/include/sieve
Boost:
Includes: -I/home/guntaka/src/petsc-dev-xlc/externalpackages/Boost/
c2html:
sowing:
PETSc:
PETSC_ARCH: linux-gnu-cxx-opt
PETSC_DIR: /home/guntaka/src/petsc-dev-xlc
Clanguage: Cxx
Scalar type: real
Precision: double
Memory alignment: 16
shared libraries: disabled
dynamic libraries: disabled
xxx=========================================================================xxx
Configure stage complete. Now build PETSc libraries with:
make PETSC_DIR=/home/guntaka/src/petsc-dev-xlc PETSC_ARCH=linux-gnu-cxx-opt all
xxx=========================================================================xxx
Note the PETSC_ARCH was a result of existing environment variable. It could be different.
Build of petsc-dec seems to be fine. But there is problem compiling source code:
"/home/guntaka/src/petsc-dev-xlc/include/sieve/Mesh.hh", line 110.30: 1540-1110 (S) The referenced type "ALE::Bundle" contains a circular reference back to "".
petsc-dev with preinstalled blas-lapack
Ref: http://www.cisl.ucar.edu/docs/frost/libs.jsp
./configure —with-clanguage=C++ —with-sieve=1 —with-opt-sieve=1 —download-chaco=1 —download-boost=1 —with-shared=0 —with-debugging=no —with-mpi-shared=0 —with-batch —with-cc=mpxlc —with-cxx=mpxlC —with-fc=mpxlf —with-blas=/contrib/bgl/lib/libblas440.a —with-lapack=/contrib/bgl/lib/liblapack440.a
This will not and result in the following error
Could not find a functional BLAS. Run with —with-blas-lib=<lib> to indicate the library containing BLAS. Or —download-f-blas-lapack=1 to have one automatically downloaded and installed
To enable the use of existing blas-lapack
- patch -p0 < /contrib/bgl/petsc/petsc-3.0.0-p4/petsc-3.0.0-p4.patch
- cp /contrib/bgl/petsc/petsc-3.0.0-p4/bgl-ibm-goto_lapack.py config/
- ./config/bgl-ibm-goto_lapack.py
The original will result in error
ValueError: The option —sizeof_char should probably be —sizeof-char
Apply the following patch and run it:
38,47c38,47 < '--sizeof-char=1', < '--sizeof-void-p=4', < '--sizeof-short=2', < '--sizeof-int=4', < '--sizeof-long=4', < '--sizeof-size-t=4', < '--sizeof-long-long=8', < '--sizeof-float=4', < '--sizeof-double=8', < '--bits-per-byte=8', --- > '--sizeof_char=1', > '--sizeof_void_p=4', > '--sizeof_short=2', > '--sizeof_int=4', > '--sizeof_long=4', > '--sizeof_size_t=4', > '--sizeof_long_long=8', > '--sizeof_float=4', > '--sizeof_double=8', > '--bits_per_byte=8',
- run the above PETSc configuration script
config/bgl-ibm-goto_lapack.py —with-clanguage=C++ —with-sieve=1 —with-opt-sieve=1 —download-chaco=1 —download-boost=1 —with-shared=0 —with-debugging=no —with-mpi-shared=0 —with-batch —with-cc=mpxlc —with-cxx=mpxlC —with-fc=mpxlf —with-blas=/contrib/bgl/lib/libblas440.a —with-lapack=/contrib/bgl/lib/liblapack440.a
This will not get complains about the blas and lapack.
Add another option
config/bgl-ibm-goto_lapack.py —with-clanguage=C++ —with-sieve=1 —with-opt-sieve=1 —download-chaco=1 —download-boost=1 —with-shared=0 —with-debugging=no —with-mpi-shared=0 —with-batch —with-cc=mpxlc —with-cxx=mpxlC —with-fc=mpxlf —with-blas=/contrib/bgl/lib/libblas440.a —with-lapack=/contrib/bgl/lib/liblapack440.a —CXXFLAGS="-DMPICH_IGNORE_CXX_SEEK"
Output:
Compilers:
C Compiler: mpxlc -O3 -qbgl -qarch=440d -qtune=440 -qmaxmem=-1
C++ Compiler: mpxlC -DMPICH_IGNORE_CXX_SEEK -O3 -qbgl -qarch=440d -qtune=440 -qmaxmem=-1 -+
Fortran Compiler: mpxlf -O3 -qbgl -qarch=440d -qtune=440 -qmaxmem=-1
Linkers:
Static linker: /usr/bin/ar cr
MPI:
Includes: -I/bgl/BlueLight/ppcfloor/bglsys/include
Library: -lnsl -lrt
Chaco:
Library: -Wl,-rpath,/home/guntaka/src/petsc-dev-xlc/bgl-ibm-goto-O3_440d/lib -L/home/guntaka/src/petsc-dev-xlc/bgl-ibm-goto-O3_440d/lib -lchaco
BLAS/LAPACK: -L/contrib/bgl/lib -llapack440 -L/contrib/bgl/lib -lgoto
Sieve:
Includes: -I/home/guntaka/src/petsc-dev-xlc/include/sieve
Boost:
Includes: -I/home/guntaka/src/petsc-dev-xlc/externalpackages/Boost/
c2html:
sowing:
PETSc:
PETSC_ARCH: bgl-ibm-goto-O3_440d
PETSC_DIR: /home/guntaka/src/petsc-dev-xlc
Clanguage: Cxx
Scalar type: real
Precision: double
Memory alignment: 16
shared libraries: disabled
dynamic libraries: disabled
xxx=========================================================================xxx
Configure stage complete. Now build PETSc libraries with:
make PETSC_DIR=/home/guntaka/src/petsc-dev-xlc PETSC_ARCH=bgl-ibm-goto-O3_440d all
xxx=========================================================================xxx