[ase-users] Dacapo's compilation
Tadeu Leonardo Soares e Silva
tsilva at peq.coppe.ufrj.br
Fri Mar 25 19:26:54 CET 2011
Dear Marcin
Below I have the environment used to compile on the server-Altix ICE. The
runtime of the process is taking longer than 20 minutes and remains
constant.This is independent of the number of compute nodes used.
Time spent on the Rocks cluster for the same test took about 2 to 3 minutes,
with gains by adding more compute nodes.
Could you check if there is any mistake in the libraries used for dacapo's
compilation?
Can I use netcdf4 for dacapo's compilation?
SGI Altix ICE 8200
64 CPUs Quad Core Intel Xeon 5355 (Clovertown), 2.66 GHz: 256 Cores
Nós de processamento: 32
Memória: 512 Gbytes RAM (distribuída)
Armazenamento em disco: SGI InfiniteStorage NAS (32 TBytes)
Rede: Infiniband DDR e Gigabit
Sistema operacional: Suse Linux Enterprise Server (SLES) + SGI ProPack
Compiladores: Intel e GNU (Fortran-90 e C/C++) com suporte OpenMP
MPI: MPT (SGI® Message Passing Toolkit), MVAPICH2 e OpenMPI
Steps to build the environment :
1. Build FFTW 2.1.5
./configure --prefix=$HOME/local/fftw2-2.1.5-1.intel F77=ifort CC=icc
CFLAGS=-O3 FFLAGS="-O3 -xssse3 --enable-shared --enable-threads" --with-pic
2. Build Netcdf 3.6.1
Add line "#include <string.h>" in the file netcdf-3.6.1/src/cxx/ncvalues.cpp.
export CC=icc
export CXX=icpc
export CFLAGS='-O3 -xssse3 -ip -no-prec-div -static -fpic'
export CXXFLAGS='-O3 -xssse3 -ip -no-prec-div -static -fpic'
export F77=ifort
export FC=ifort
export F90=ifort
export FFLAGS='-O3 -xssse3 -ip -no-prec-div -static -fpic'
export CPP='icc -E'
export CXXCPP='icpc -E'
export CPPFLAGS='-DNDEBUG -DpgiFortran -fpic'
./configure --prefix=$HOME/local/netcdf-3.6.1
make check
mkdir -p $HOME/local/netcdf-3.6.1
make install
3. Build Scientific Python 2.9.1
Change the file setup.py, add line "extra_compile_args.append("-fpic")"
after the line "extra_compile_args.append("-DNUMPY=1")"
export CC=icc
export CXX=icpc
export CFLAGS='-O3 -xssse3 -ip -no-prec-div -static -fpic'
export CXXFLAGS='-O3 -xssse3 -ip -no-prec-div -static -fpic'
export F77=ifort
export FC=ifort
export F90=ifort
export FFLAGS='-O3 -xssse3 -ip -no-prec-div -static -fpic'
export CPP='icc -E'
export CXXCPP='icpc -E'
export NETCDF_PREFIX=/home/users/tadeu33/local/netcdf-3.6.1
python setup.py build --compiler=intel
python setup.py install --verbose --prefix='' --
home=$HOME/local/ScientificPython-2.9.1
4. Build Dacapo
Generate dacapo_intellinux_serial.run
make intellinux
Generate dacapo_intellinux_mpi.run
make intellinux MP=mpi
4.1. Build with fftw 2.1.5
File Makefile from directory src :
INTELLINUX_OPT = -O3 -xssse3
# LIBS_intellinux=-L${NETCDF}/lib -lnetcdf -L${MKLPATH} -lfftw2xf_intel -
lmkl_lapack -lmkl_ia32 -lguide -lpthread
LIBS_intellinux= ${BLASLAPACK} ${NETCDF}/libnetcdf.a ${FFTW}/libfftw.a
Environment variables :
BLASLAPACK='-L/sw/intel/Compiler/11.1/072/mkl/lib/em64t -lmkl_lapack -
lmkl_intel_lp64 -lmkl_core -lguide -lpthread -lmkl_intel_thread'
NETCDF=/home/users/tadeu33/local/netcdf-3.6.1/lib
FFTW=/home/users/tadeu33/local/fftw2-2.1.5-1.intel/lib
MPIDIR=/sw/mpi/intel/openmpi-1.4.2
MPI_LIBDIR=${MPIDIR}/lib64
MPI_BINDIR=${MPIDIR}/bin
MPI_INCLUDEDIR=${MPIDIR}/include
export BLASLAPACK NETCDF FFTW MPIDIR MPI_LIBDIR MPI_BINDIR MPI_INCLUDEDIR
4.2. Build with fftw from [MKL]
File Makefile from directory src :
INTELLINUX_OPT = -O3 -xssse3
# LIBS_intellinux=-L${NETCDF}/lib -lnetcdf -L${MKLPATH} -lfftw2xf_intel -
lmkl_lapack -lmkl_ia32 -lguide -lpthread
LIBS_intellinux= ${BLASLAPACK} ${NETCDF}/libnetcdf.a
${FFTW}/libfftw2xf_intel.a
Environment:
BLASLAPACK='-L/sw/intel/Compiler/11.1/072/mkl/lib/em64t -lmkl_lapack -
lmkl_intel_lp64 -lmkl_core -lguide -lpthread -lmkl_intel_thread'
NETCDF=/home/users/tadeu33/local/netcdf-3.6.1/lib
FFTW=/home/users/tadeu33/local/fftw2-2.1.5-1.intel/lib
MPIDIR=/sw/mpi/intel/openmpi-1.4.2
MPI_LIBDIR=${MPIDIR}/lib64
MPI_BINDIR=${MPIDIR}/bin
MPI_INCLUDEDIR=${MPIDIR}/include
export BLASLAPACK NETCDF FFTW MPIDIR MPI_LIBDIR MPI_BINDIR MPI_INCLUDEDIR
Bibliotecas do [MKL] :
tadeu33 at service0:~/work_20110323/dacapo/src>
ls /sw/intel/Compiler/11.1/072/mkl/lib/em64t/
libfftw2xf_intel.a libmkl_blacs_sgimpt_ilp64.a
libmkl_gf_lp64.a libmkl_intel_thread.a
libmkl_scalapack_ilp64.a libmkl_vml_avx.so
libmkl_avx.so libmkl_blacs_sgimpt_lp64.a
libmkl_gf_lp64.so libmkl_intel_thread.so
libmkl_scalapack_ilp64.so libmkl_vml_def.so
libmkl_blacs_ilp64.a libmkl_blas95_ilp64.a
libmkl_gnu_thread.a libmkl_lapack95_ilp64.a
libmkl_scalapack_lp64.a libmkl_vml_mc2.so
libmkl_blacs_intelmpi_ilp64.a libmkl_blas95_lp64.a
libmkl_gnu_thread.so libmkl_lapack95_lp64.a
libmkl_scalapack_lp64.so libmkl_vml_mc3.so
libmkl_blacs_intelmpi_ilp64.so libmkl_cdft_core.a
libmkl_intel_ilp64.a libmkl_lapack.so
libmkl_sequential.a libmkl_vml_mc.so
libmkl_blacs_intelmpi_lp64.a libmkl_core.a
libmkl_intel_ilp64.so libmkl_mc3.so
libmkl_sequential.so libmkl_vml_p4n.so
libmkl_blacs_intelmpi_lp64.so libmkl_core.so
libmkl_intel_lp64.a libmkl_mc.so
libmkl_solver_ilp64.a locale
libmkl_blacs_lp64.a libmkl_def.so
libmkl_intel_lp64.so libmkl_p4n.so
libmkl_solver_ilp64_sequential.a
libmkl_blacs_openmpi_ilp64.a libmkl_gf_ilp64.a
libmkl_intel_sp2dp.a libmkl_pgi_thread.a libmkl_solver_lp64.a
libmkl_blacs_openmpi_lp64.a libmkl_gf_ilp64.so
libmkl_intel_sp2dp.so libmkl_pgi_thread.so
libmkl_solver_lp64_sequential.a
Enviroment Build :
tadeu33 at service0:~> module list
Currently Loaded Modulefiles:
1) openmpi/1.4.2-intel(default) 3) torque
2) gotoblas2/1.13(default) 4) intel/11.1.072
tadeu33 at service0:~> module display openmpi/1.4.2-intel
-------------------------------------------------------------------
/sw/modulefiles/openmpi/1.4.2-intel:
module-whatis Sets up Open MPI in your enviornment
prepend-path PATH /sw/mpi/intel/openmpi-1.4.2/bin
prepend-path C_INCLUDE_PATH /sw/mpi/intel/openmpi-1.4.2/include
prepend-path CPLUS_INCLUDE_PATH /sw/mpi/intel/openmpi-1.4.2/include
prepend-path FPATH /sw/mpi/intel/openmpi-1.4.2/include
prepend-path LD_LIBRARY_PATH /sw/mpi/intel/openmpi-1.4.2/lib64
prepend-path LIBRARY_PATH /sw/mpi/intel/openmpi-1.4.2/lib64
prepend-path MANPATH /sw/mpi/intel/openmpi-1.4.2/share/man
-------------------------------------------------------------------
tadeu33 at service0:~> module display intel/11.1.072
-------------------------------------------------------------------
/sw/modulefiles/intel/11.1.072:
module-whatis Intel Compiler
module-whatis Category: compiler, runtime support
module-whatis Description: Intel Compiler Family (C/C++/Fortran for
x86_64)
module-whatis URL: http://www.intel.com/cd/software/products/asmo-
na/eng/compilers/284132.htm
prepend-path NLSPATH /sw/intel/Compiler/11.1/072/idb/intel64/locale/%l_%
t/%N
prepend-path CPATH /sw/intel/Compiler/11.1/072/tbb/include
prepend-path
LIBRARY_PATH /sw/intel/Compiler/11.1/072/tbb/em64t/cc4.1.0_libc2.4_kernel2.6.
16.21/lib
prepend-path
LD_LIBRARY_PATH /sw/intel/Compiler/11.1/072/tbb/em64t/cc4.1.0_libc2.4_kernel2
.6.16.21/lib
prepend-path MKLROOT /sw/intel/Compiler/11.1/072/mkl
prepend-path INCLUDE /sw/intel/Compiler/11.1/072/mkl/include
prepend-path CPATH /sw/intel/Compiler/11.1/072/mkl/include
prepend-path FPATH /sw/intel/Compiler/11.1/072/mkl/include
prepend-path LIBRARY_PATH /sw/intel/Compiler/11.1/072/mkl/lib/em64t
prepend-path LD_LIBRARY_PATH /sw/intel/Compiler/11.1/072/mkl/lib/em64t
prepend-path MANPATH /sw/intel/Compiler/11.1/072/mkl/man/en_US
prepend-path NLSPATH /sw/intel/Compiler/11.1/072/mkl/lib/em64t/locale/%
l_%t/%N
prepend-path IPPROOT /sw/intel/Compiler/11.1/072/ipp/em64t
prepend-path INCLUDE /sw/intel/Compiler/11.1/072/ipp/em64t/include
prepend-path CPATH /sw/intel/Compiler/11.1/072/ipp/em64t/include
prepend-path LIB /sw/intel/Compiler/11.1/072/ipp/em64t/lib
prepend-path LIBRARY_PATH /sw/intel/Compiler/11.1/072/ipp/em64t/lib
prepend-path
LD_LIBRARY_PATH /sw/intel/Compiler/11.1/072/ipp/em64t/sharedlib
prepend-path NLSPATH /sw/intel/Compiler/11.1/072/ipp/em64t/lib/locale/%
l_%t/%N
prepend-path PATH /sw/intel/Compiler/11.1/072/bin/intel64
prepend-path LIBRARY_PATH /sw/intel/Compiler/11.1/072/lib/intel64
prepend-path LD_LIBRARY_PATH /sw/intel/Compiler/11.1/072/lib/intel64
prepend-path MANPATH /sw/intel/Compiler/11.1/072/man/en_US
prepend-path NLSPATH /sw/intel/Compiler/11.1/072/lib/intel64/locale/%l_%
t/%N
-------------------------------------------------------------------
Environment that can be add :
tadeu33 at service0:~> module avail
------------------------------- /sw/modulefiles -----------------------------
---
gotoblas2/1.13(default) mvapich2/1.5.1-p1-gcc
intel/11.1.064 mvapich2/1.5.1-p1-intel
intel/11.1.069 netcdf4
intel/11.1.072(default) openmpi/1.4-gcc
java/1.6.0_21 openmpi/1.4-intel
java/1.6.0_23 openmpi/1.4.1-gcc
java/1.6.0_24(default) openmpi/1.4.1-intel
maui openmpi/1.4.2-gcc
mpj/0.35 openmpi/1.4.2-intel(default)
mpt/2.02(default) openmpi/1.4.3-gcc
mvapich2/1.4-gcc openmpi/1.4.3-intel
mvapich2/1.4-intel papi/4.1.0
mvapich2/1.4.1-gcc papi/4.1.2.1(default)
mvapich2/1.4.1-intel perfboost
mvapich2/1.5-gcc torque
mvapich2/1.5-intel(default)
Netcdf4 environment variables:
tadeu33 at service0:~> module display netcdf4
-------------------------------------------------------------------
/sw/modulefiles/netcdf4:
module-whatis Sets up the NetCDF environment.
append-path PATH /sw/shared/netcdf/4.0.1/bin
append-path LIBRARY_PATH /sw/shared/netcdf/4.0.1/lib
append-path LD_LIBRARY_PATH /sw/shared/netcdf/4.0.1/lib
setenv NETCDFC_LD -L/sw/shared/netcdf/4.0.1/lib -lnetcdf
setenv NETCDFF_LD -L/sw/shared/netcdf/4.0.1/lib -lnetcdf
setenv NETCDFCPP_LD -L/sw/shared/netcdf/4.0.1/lib -lnetcdf_c++
setenv NETCDF_INC -I/sw/shared/netcdf/4.0.1/include
setenv NETCDF_INC_F90 -I/sw/shared/netcdf/4.0.1/include
-------------------------------------------------------------------
Sincerely,
Tadeu Leonardo
+++++++++++++++++++++++++++++++++++++++++++++++++++++
PEQ/COPPE renova o nivel de curso 7, maximo na CAPES:
45 anos de excelencia no ensino e pesquisa de pos-graduação em
Engenharia Quimica.
************************************
PEQ/COPPE : 45 years of commitment to excellence in teaching and
research in Chemical Engineering.
+++++++++++++++++++++++++++++++++++++++++++++++++++++
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Dacapo's compilation.doc
Type: application/msword
Size: 63488 bytes
Desc: not available
URL: <http://listserv.fysik.dtu.dk/pipermail/ase-users/attachments/20110325/874a9356/attachment.doc>
More information about the ase-users
mailing list