Frost

Introduction

Login

With SSH key only. Need to setup the key pair first.

Compiler Version

Ref: http://www.cisl.ucar.edu/docs/frost/cbr.jsp#compiling
Note:

mpxlc=mpixlc

guntaka@fr0103en:~/src/petsc-3.1-p0> mpicxx -show
/bgl/BlueLight/V1R3M4_300_2008-080728/ppc/blrts-gnu/bin/powerpc-bgl-blrts-gnu-g++ -I/bgl/BlueLight/V1R3M4_300_2008-080728/ppc/bglsys/include -L/bgl/BlueLight/V1R3M4_300_2008-080728/ppc/bglsys/lib -lcxxmpich.rts -lmpich.rts -L/bgl/BlueLight/V1R3M4_300_2008-080728/ppc/bglsys/lib -lmsglayer.rts -ldevices.rts -lrts.rts -ldevices.rts -lrts.rts

guntaka@fr0103en:~/src/petsc-3.1-p0> /bgl/BlueLight/V1R3M4_300_2008-080728/ppc/blrts-gnu/bin/powerpc-bgl-blrts-gnu-g++ --version
powerpc-bgl-blrts-gnu-g++ (GCC) 3.4.3
Copyright (C) 2004 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Cobalt Scheduler

Ref: http://www.cisl.ucar.edu/docs/frost/cbr.jsp#Submission

  • cqstat -q: see a list of queues
guntaka@fr0103en:~/src/petsc-3.1-p0> cqstat -q
Name                   State    Users                                      MinTime  MaxTime   MaxRunning  MaxQueued  MaxUserNodes  TotalNodes  Priority  
=========================================================================================================================================================
default                running  None                                       None     24:00:00  12          20         None          None        0         
debug                  running  None                                       None     02:00:00  8           20         None          None        0
  • cqsub -n <power of 2> -t <h:m:s> <program>

PETSc

C version without Sieve

The simplest case:

  • ./configure —with-batch —download-f-blas-lapack=1
  • cqsub -q debug -n 1 -t 00:10:00 conftest
  • ./reconfigure.py

output

Compilers:
  C Compiler:         mpicc  -Wall -Wwrite-strings -Wno-strict-aliasing -g3 
  Fortran Compiler:   mpif77  -Wall -Wno-unused-variable -g  
Linkers:
  Static linker:   /usr/bin/ar cr
MPI:
  Includes: -I/bgl/BlueLight/V1R3M4_300_2008-080728/ppc/bglsys/include
BLAS/LAPACK: -Wl,-rpath,/home/guntaka/src/petsc-3.1-p0/linux-gnu-c-debug/lib -L/home/guntaka/src/petsc-3.1-p0/linux-gnu-c-debug/lib -lflapack -Wl,-rpath,/home/guntaka/src/petsc-3.1-p0/linux-gnu-c-debug/lib -L/home/guntaka/src/petsc-3.1-p0/linux-gnu-c-debug/lib -lfblas
PETSc:
  PETSC_ARCH: linux-gnu-c-debug
  PETSC_DIR: /home/guntaka/src/petsc-3.1-p0
  Clanguage: C
  Scalar type: real
  Precision: double
  shared libraries: disabled
  dynamic libraries: disabled
  Memory alignment: 16
xxx=========================================================================xxx
   Configure stage complete. Now build PETSc libraries with:
   make PETSC_DIR=/home/guntaka/src/petsc-3.1-p0 PETSC_ARCH=linux-gnu-c-debug all
xxx=========================================================================xxx

One could also get it to work with mpxlc

C++ Version with Sieve

  • ./configure —with-clanguage=C++ —with-sieve=1 —with-opt-sieve=1 —download-chaco=1 —download-boost=1 —with-shared=0 —with-debugging=no —with-mpi-shared=0 —download-f-blas-lapack=1 —with-batch
  • cqsub -q debug -n 1 -t 00:10:00 conftest. Make sure the job is finished by "cqstat".
  • ./reconfigure.py

Output:

Compilers:
  C Compiler:         mpicc  -Wall -Wwrite-strings -Wno-strict-aliasing -O 
  C++ Compiler:       mpicxx  -Wall -Wwrite-strings -Wno-strict-aliasing -O   
  Fortran Compiler:   mpif77  -Wall -Wno-unused-variable -O  
Linkers:
  Static linker:   /usr/bin/ar cr
MPI:
  Includes: -I/bgl/BlueLight/V1R3M4_300_2008-080728/ppc/bglsys/include
BLAS/LAPACK: -Wl,-rpath,/home/guntaka/src/petsc-3.1-p0/linux-gnu-cxx-opt/lib -L/home/guntaka/src/petsc-3.1-p0/linux-gnu-cxx-opt/lib -lflapack -Wl,-rpath,/home/guntaka/src/petsc-3.1-p0/linux-gnu-cxx-opt/lib -L/home/guntaka/src/petsc-3.1-p0/linux-gnu-cxx-opt/lib -lfblas
Chaco:
  Library:  -Wl,-rpath,/home/guntaka/src/petsc-3.1-p0/linux-gnu-cxx-opt/lib -L/home/guntaka/src/petsc-3.1-p0/linux-gnu-cxx-opt/lib -lchaco
Sieve:
  Includes: -I/home/guntaka/src/petsc-3.1-p0/include/sieve
Boost:
  Includes: -I/home/guntaka/src/petsc-3.1-p0/externalpackages/Boost/ -I/bgl/BlueLight/V1R3M4_300_2008-080728/ppc/bglsys/include
PETSc:
  PETSC_ARCH: linux-gnu-cxx-opt
  PETSC_DIR: /home/guntaka/src/petsc-3.1-p0
  Clanguage: Cxx
  Scalar type: real
  Precision: double
  shared libraries: disabled
  dynamic libraries: disabled
  Memory alignment: 16
xxx=========================================================================xxx
   Configure stage complete. Now build PETSc libraries with:
   make PETSC_DIR=/home/guntaka/src/petsc-3.1-p0 PETSC_ARCH=linux-gnu-cxx-opt all
xxx=========================================================================xxx

make error

/home/guntaka/src/petsc-3.1-p0/include/sieve/ALE_args.hh: In member function `ALE::ArgValue ALE::ArgDB::operator[](const string&) const':
/home/guntaka/src/petsc-3.1-p0/include/sieve/ALE_args.hh:132: error: call of overloaded `ArgValue(ALE::ArgValue)' is ambiguous
/home/guntaka/src/petsc-3.1-p0/include/sieve/ALE_args.hh:64: note: candidates are: ALE::ArgValue::ArgValue(const ALE::ArgValue&)
/home/guntaka/src/petsc-3.1-p0/include/sieve/ALE_args.hh:67: note:                 ALE::ArgValue::ArgValue(const boost::program_options::variable_value&)
/home/guntaka/src/petsc-3.1-p0/include/sieve/ALE_args.hh:132: error:   initializing temporary from result of `ALE::ArgValue::ArgValue(const boost::program_options::variable_value&)'

From Matt:
this is a bug in the IBM compiler version I have had other places. In the latest
petsc-dev, I just commented out the #include of ALE_args.hh in ALE.hh. This fixes
it. Everyone hates compiler bugs.

Solution: use petsc-dev.

petsc-dev with sieve

mercurial

PYTHON=python2.5

  • vi ~/.bashrc
alias python=python2.5
export PYTHONPATH=/home/guntaka/lib/python2.5/site-packages
export PATH=$PATH:$HOME/bin

petsc-dev

The same as above.

vi ~/.bashrc

export PETSC_DIR=/home/guntaka/src/petsc-dev
export PETSC_ARCH=linux-gnu-cxx-opt

fcpu

All the .cxx codes are compiled successfully into objects. But there is an error at the final linking stage:

mpicxx -Wall -Wwrite-strings -Wno-strict-aliasing -O  -o fcpu main.o Options.o USG.o Wall.o PlaneWall.o BC.o Quadrature.o Tetrahedron.o TetrahedronTaylorHood.o Cylinder.o Sphere.o CylinderWall.o CFD.o NSSolver.o ParticulateFlow.o BodyFitMeshPF.o CollisionBox.o GranularFlow.o -Wl,-rpath,/home/guntaka/src/petsc-dev/linux-gnu-cxx-opt/lib -L/home/guntaka/src/petsc-dev/linux-gnu-cxx-opt/lib -lpetsc    -Wl,-rpath,/home/guntaka/src/petsc-dev/linux-gnu-cxx-opt/lib -L/home/guntaka/src/petsc-dev/linux-gnu-cxx-opt/lib -lchaco -lflapack -lfblas -lm -L/bgl/BlueLight/V1R3M4_300_2008-080728/ppc/bglsys/lib -L/bgl/BlueLight/V1R3M4_300_2008-080728/ppc/blrts-gnu/lib/gcc/powerpc-bgl-blrts-gnu/3.4.3 -L/bgl/BlueLight/V1R3M4_300_2008-080728/ppc/blrts-gnu/powerpc-bgl-blrts-gnu/lib -lmpich.rts -lmsglayer.rts -ldevices.rts -lrts.rts -lg2c -lm -lm -lcxxmpich.rts -lstdc++ -lmpich.rts -lmsglayer.rts -ldevices.rts -lrts.rts  -Wl,-rpath,/home/guntaka/src/petsc-dev/linux-gnu-cxx-opt/lib -L/home/guntaka/src/petsc-dev/linux-gnu-cxx-opt/lib -lpetsc   -Wl,-rpath,/home/guntaka/src/petsc-dev/linux-gnu-cxx-opt/lib -L/home/guntaka/src/petsc-dev/linux-gnu-cxx-opt/lib -lchaco -lflapack -lfblas -lm -L/bgl/BlueLight/V1R3M4_300_2008-080728/ppc/bglsys/lib -L/bgl/BlueLight/V1R3M4_300_2008-080728/ppc/blrts-gnu/lib/gcc/powerpc-bgl-blrts-gnu/3.4.3 -L/bgl/BlueLight/V1R3M4_300_2008-080728/ppc/blrts-gnu/powerpc-bgl-blrts-gnu/lib -lmpich.rts -lmsglayer.rts -ldevices.rts -lrts.rts -lg2c -lm -lm -lcxxmpich.rts -lstdc++ -lmpich.rts -lmsglayer.rts -ldevices.rts -lrts.rts 
collect2: ld terminated with signal 11 [Segmentation fault]
make: *** [fcpu] Error 1

petsc-dev-sieve with IBM compilers

./configure —with-clanguage=C++ —with-sieve=1 —with-opt-sieve=1 —download-chaco=1 —download-boost=1 —with-shared=0 —with-debugging=no —with-mpi-shared=0 —download-f-blas-lapack=1 —with-batch —with-cc=mpxlc —with-cxx=mpxlC —with-fc=mpxlf

Note: —with-fc=mpxlf90 does not work (in building blas-lapack).

cqsub -q debug -n 1 -t 00:10:00 conftest
./reconfigure.py

output:

Compilers:
  C Compiler:         mpxlc  -O 
  C++ Compiler:       mpxlC  -qrtti=dyna -O -+  
  Fortran Compiler:   mpxlf  -O  
Linkers:
  Static linker:   /usr/bin/ar cr
MPI:
  Library:  -lnsl -lrt
Chaco:
  Library:  -Wl,-rpath,/home/guntaka/src/petsc-dev-xlc/linux-gnu-cxx-opt/lib -L/home/guntaka/src/petsc-dev-xlc/linux-gnu-cxx-opt/lib -lchaco
BLAS/LAPACK: -Wl,-rpath,/home/guntaka/src/petsc-dev-xlc/linux-gnu-cxx-opt/lib -L/home/guntaka/src/petsc-dev-xlc/linux-gnu-cxx-opt/lib -lflapack -Wl,-rpath,/home/guntaka/src/petsc-dev-xlc/linux-gnu-cxx-opt/lib -L/home/guntaka/src/petsc-dev-xlc/linux-gnu-cxx-opt/lib -lfblas
Sieve:
  Includes: -I/home/guntaka/src/petsc-dev-xlc/include/sieve
Boost:
  Includes: -I/home/guntaka/src/petsc-dev-xlc/externalpackages/Boost/
c2html:
sowing:
PETSc:
  PETSC_ARCH: linux-gnu-cxx-opt
  PETSC_DIR: /home/guntaka/src/petsc-dev-xlc
  Clanguage: Cxx
  Scalar type: real
  Precision: double
  Memory alignment: 16
  shared libraries: disabled
  dynamic libraries: disabled
xxx=========================================================================xxx
   Configure stage complete. Now build PETSc libraries with:
   make PETSC_DIR=/home/guntaka/src/petsc-dev-xlc PETSC_ARCH=linux-gnu-cxx-opt all
xxx=========================================================================xxx

Note the PETSC_ARCH was a result of existing environment variable. It could be different.

Build of petsc-dec seems to be fine. But there is problem compiling source code:

"/home/guntaka/src/petsc-dev-xlc/include/sieve/Mesh.hh", line 110.30: 1540-1110 (S) The referenced type "ALE::Bundle" contains a circular reference back to "".

petsc-dev with preinstalled blas-lapack

Ref: http://www.cisl.ucar.edu/docs/frost/libs.jsp

./configure —with-clanguage=C++ —with-sieve=1 —with-opt-sieve=1 —download-chaco=1 —download-boost=1 —with-shared=0 —with-debugging=no —with-mpi-shared=0 —with-batch —with-cc=mpxlc —with-cxx=mpxlC —with-fc=mpxlf —with-blas=/contrib/bgl/lib/libblas440.a —with-lapack=/contrib/bgl/lib/liblapack440.a

This will not and result in the following error

Could not find a functional BLAS. Run with —with-blas-lib=<lib> to indicate the library containing BLAS. Or —download-f-blas-lapack=1 to have one automatically downloaded and installed

To enable the use of existing blas-lapack

  • patch -p0 < /contrib/bgl/petsc/petsc-3.0.0-p4/petsc-3.0.0-p4.patch
  • cp /contrib/bgl/petsc/petsc-3.0.0-p4/bgl-ibm-goto_lapack.py config/
  • ./config/bgl-ibm-goto_lapack.py

The original will result in error

ValueError: The option —sizeof_char should probably be —sizeof-char

Apply the following patch and run it:

38,47c38,47
<   '--sizeof-char=1',
<   '--sizeof-void-p=4',
<   '--sizeof-short=2',
<   '--sizeof-int=4',
<   '--sizeof-long=4',
<   '--sizeof-size-t=4',
<   '--sizeof-long-long=8',
<   '--sizeof-float=4',
<   '--sizeof-double=8',
<   '--bits-per-byte=8',
---
>   '--sizeof_char=1',
>   '--sizeof_void_p=4',
>   '--sizeof_short=2',
>   '--sizeof_int=4',
>   '--sizeof_long=4',
>   '--sizeof_size_t=4',
>   '--sizeof_long_long=8',
>   '--sizeof_float=4',
>   '--sizeof_double=8',
>   '--bits_per_byte=8',
  • run the above PETSc configuration script

config/bgl-ibm-goto_lapack.py —with-clanguage=C++ —with-sieve=1 —with-opt-sieve=1 —download-chaco=1 —download-boost=1 —with-shared=0 —with-debugging=no —with-mpi-shared=0 —with-batch —with-cc=mpxlc —with-cxx=mpxlC —with-fc=mpxlf —with-blas=/contrib/bgl/lib/libblas440.a —with-lapack=/contrib/bgl/lib/liblapack440.a

This will not get complains about the blas and lapack.
Add another option

config/bgl-ibm-goto_lapack.py —with-clanguage=C++ —with-sieve=1 —with-opt-sieve=1 —download-chaco=1 —download-boost=1 —with-shared=0 —with-debugging=no —with-mpi-shared=0 —with-batch —with-cc=mpxlc —with-cxx=mpxlC —with-fc=mpxlf —with-blas=/contrib/bgl/lib/libblas440.a —with-lapack=/contrib/bgl/lib/liblapack440.a —CXXFLAGS="-DMPICH_IGNORE_CXX_SEEK"

Output:

Compilers:
  C Compiler:         mpxlc  -O3 -qbgl -qarch=440d -qtune=440 -qmaxmem=-1 
  C++ Compiler:       mpxlC -DMPICH_IGNORE_CXX_SEEK -O3 -qbgl -qarch=440d -qtune=440 -qmaxmem=-1 -+  
  Fortran Compiler:   mpxlf  -O3 -qbgl -qarch=440d -qtune=440 -qmaxmem=-1  
Linkers:
  Static linker:   /usr/bin/ar cr
MPI:
  Includes: -I/bgl/BlueLight/ppcfloor/bglsys/include
  Library:  -lnsl -lrt
Chaco:
  Library:  -Wl,-rpath,/home/guntaka/src/petsc-dev-xlc/bgl-ibm-goto-O3_440d/lib -L/home/guntaka/src/petsc-dev-xlc/bgl-ibm-goto-O3_440d/lib -lchaco
BLAS/LAPACK: -L/contrib/bgl/lib -llapack440 -L/contrib/bgl/lib -lgoto
Sieve:
  Includes: -I/home/guntaka/src/petsc-dev-xlc/include/sieve
Boost:
  Includes: -I/home/guntaka/src/petsc-dev-xlc/externalpackages/Boost/
c2html:
sowing:
PETSc:
  PETSC_ARCH: bgl-ibm-goto-O3_440d
  PETSC_DIR: /home/guntaka/src/petsc-dev-xlc
  Clanguage: Cxx
  Scalar type: real
  Precision: double
  Memory alignment: 16
  shared libraries: disabled
  dynamic libraries: disabled
xxx=========================================================================xxx
   Configure stage complete. Now build PETSc libraries with:
   make PETSC_DIR=/home/guntaka/src/petsc-dev-xlc PETSC_ARCH=bgl-ibm-goto-O3_440d all
xxx=========================================================================xxx

References

Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License