[ase-users] Dacapo's compilation
Marcin Dulak
Marcin.Dulak at fysik.dtu.dk
Fri Mar 25 22:12:44 CET 2011
Hi,
You can use netcdf 4 with dacapo.
Run ldd command on your dacapo executable to make sure that it's linked
correctly, for example you may see something like this:
$ldd
/opt/campos-dacapo/2.7.16/1.el5.fys.gfortran43.4.3.2.openmpi.1.3.3.mkl.10.1.3.027.mkl_lapack.10.1.3.027.mkl/bin/dacapo_2.7.16-1_mpi.run
libguide.so => /opt/intel/mkl/10.1.3.027/lib/em64t/libguide.so
(0x00002ab8c62c7000)
libmkl_intel_lp64.so =>
/opt/intel/mkl/10.1.3.027/lib/em64t/libmkl_intel_lp64.so
(0x00002ab8c643a000)
libmkl_intel_thread.so =>
/opt/intel/mkl/10.1.3.027/lib/em64t/libmkl_intel_thread.so
(0x00002ab8c67b0000)
libmkl_core.so =>
/opt/intel/mkl/10.1.3.027/lib/em64t/libmkl_core.so (0x00002ab8c7515000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x000000328aa00000)
libmkl_lapack.so =>
/opt/intel/mkl/10.1.3.027/lib/em64t/libmkl_lapack.so (0x00002ab8c7724000)
libmpi_f90.so.0 =>
/opt/openmpi/1.3.3-1.el5.fys.gfortran43.4.3.2/lib64/libmpi_f90.so.0
(0x00002ab8c8083000)
libmpi_f77.so.0 =>
/opt/openmpi/1.3.3-1.el5.fys.gfortran43.4.3.2/lib64/libmpi_f77.so.0
(0x00002ab8c8286000)
libmpi.so.0 =>
/opt/openmpi/1.3.3-1.el5.fys.gfortran43.4.3.2/lib64/libmpi.so.0
(0x00002ab8c84b8000)
libopen-rte.so.0 =>
/opt/openmpi/1.3.3-1.el5.fys.gfortran43.4.3.2/lib64/libopen-rte.so.0
(0x00002ab8c8761000)
libopen-pal.so.0 =>
/opt/openmpi/1.3.3-1.el5.fys.gfortran43.4.3.2/lib64/libopen-pal.so.0
(0x00002ab8c89a8000)
libdl.so.2 => /lib64/libdl.so.2 (0x000000328a600000)
libnsl.so.1 => /lib64/libnsl.so.1 (0x000000328da00000)
libutil.so.1 => /lib64/libutil.so.1 (0x0000003297c00000)
libgfortran.so.3 => /usr/lib64/libgfortran.so.3 (0x00002ab8c8c1c000)
libm.so.6 => /lib64/libm.so.6 (0x000000328a200000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x000000329a000000)
libc.so.6 => /lib64/libc.so.6 (0x0000003289e00000)
/lib64/ld-linux-x86-64.so.2 (0x0000003289a00000)
About testing: first let's make clear what is the test job.
Can you run on both systems (server-Altix ICE, and the Rocks cluster),
in serial,
the CO molecule from
http://gilgamesh.cheme.cmu.edu/doc/software/jacapo/1-molecules/1.2-calculating-simple-properties/1.2.1-energy-and-forces/1.2.1.0-energy-and-forces.html,
setting pw to let's say 750.
pw needs to be increased to make the runtime longer, so less influenced
by irregularities inherent to standard Linux systems.
Run the job several times, and take the fastest run as the result.
You use intel compiler+mkl so check that you don't run in threaded mode
(see bottom of
https://wiki.fysik.dtu.dk/gpaw/install/Linux/Niflheim/Niflheim.html)
Make sure that RAM memory is not exceeded, and you have exclusive access
to the node (no other jobs running). From that we will know what is the
relative performance of the two systems.
Then verify that you are able to run your jobs in parallel (see my
comments on the use of mpiexec/mpirun here
https://listserv.fysik.dtu.dk/pipermail/ase-users/2011-March/000975.html).
Analyse a bulk system, for example
http://gilgamesh.cheme.cmu.edu/doc/software/jacapo/2-bulk/2.2-unitcell-optimization/2.2.0-unit-cell-optimization.html#lattice-constant-and-bulk-modulus,
this should scale with the increasing number of cores used (again,
increase pw bit, so runtime gets longer).
Best regards,
Marcin
Tadeu Leonardo Soares e Silva wrote:
> Dear Marcin
>
> Below I have the environment used to compile on the server-Altix ICE. The
> runtime of the process is taking longer than 20 minutes and remains
> constant.This is independent of the number of compute nodes used.
>
> Time spent on the Rocks cluster for the same test took about 2 to 3 minutes,
> with gains by adding more compute nodes.
>
> Could you check if there is any mistake in the libraries used for dacapo's
> compilation?
>
> Can I use netcdf4 for dacapo's compilation?
>
> SGI Altix ICE 8200
>
> 64 CPUs Quad Core Intel Xeon 5355 (Clovertown), 2.66 GHz: 256 Cores
> Nós de processamento: 32
> Memória: 512 Gbytes RAM (distribuída)
> Armazenamento em disco: SGI InfiniteStorage NAS (32 TBytes)
> Rede: Infiniband DDR e Gigabit
> Sistema operacional: Suse Linux Enterprise Server (SLES) + SGI ProPack
> Compiladores: Intel e GNU (Fortran-90 e C/C++) com suporte OpenMP
> MPI: MPT (SGI® Message Passing Toolkit), MVAPICH2 e OpenMPI
>
>
> Steps to build the environment :
>
> 1. Build FFTW 2.1.5
> ./configure --prefix=$HOME/local/fftw2-2.1.5-1.intel F77=ifort CC=icc
> CFLAGS=-O3 FFLAGS="-O3 -xssse3 --enable-shared --enable-threads" --with-pic
>
> 2. Build Netcdf 3.6.1
> Add line "#include <string.h>" in the file netcdf-3.6.1/src/cxx/ncvalues.cpp.
>
> export CC=icc
> export CXX=icpc
> export CFLAGS='-O3 -xssse3 -ip -no-prec-div -static -fpic'
> export CXXFLAGS='-O3 -xssse3 -ip -no-prec-div -static -fpic'
> export F77=ifort
> export FC=ifort
> export F90=ifort
> export FFLAGS='-O3 -xssse3 -ip -no-prec-div -static -fpic'
> export CPP='icc -E'
> export CXXCPP='icpc -E'
> export CPPFLAGS='-DNDEBUG -DpgiFortran -fpic'
> ./configure --prefix=$HOME/local/netcdf-3.6.1
> make check
> mkdir -p $HOME/local/netcdf-3.6.1
> make install
>
> 3. Build Scientific Python 2.9.1
>
> Change the file setup.py, add line "extra_compile_args.append("-fpic")"
> after the line "extra_compile_args.append("-DNUMPY=1")"
>
> export CC=icc
> export CXX=icpc
> export CFLAGS='-O3 -xssse3 -ip -no-prec-div -static -fpic'
> export CXXFLAGS='-O3 -xssse3 -ip -no-prec-div -static -fpic'
> export F77=ifort
> export FC=ifort
> export F90=ifort
> export FFLAGS='-O3 -xssse3 -ip -no-prec-div -static -fpic'
> export CPP='icc -E'
> export CXXCPP='icpc -E'
> export NETCDF_PREFIX=/home/users/tadeu33/local/netcdf-3.6.1
> python setup.py build --compiler=intel
> python setup.py install --verbose --prefix='' --
> home=$HOME/local/ScientificPython-2.9.1
>
> 4. Build Dacapo
>
> Generate dacapo_intellinux_serial.run
> make intellinux
>
> Generate dacapo_intellinux_mpi.run
> make intellinux MP=mpi
>
> 4.1. Build with fftw 2.1.5
> File Makefile from directory src :
>
> INTELLINUX_OPT = -O3 -xssse3
> # LIBS_intellinux=-L${NETCDF}/lib -lnetcdf -L${MKLPATH} -lfftw2xf_intel -
> lmkl_lapack -lmkl_ia32 -lguide -lpthread
> LIBS_intellinux= ${BLASLAPACK} ${NETCDF}/libnetcdf.a ${FFTW}/libfftw.a
>
> Environment variables :
> BLASLAPACK='-L/sw/intel/Compiler/11.1/072/mkl/lib/em64t -lmkl_lapack -
> lmkl_intel_lp64 -lmkl_core -lguide -lpthread -lmkl_intel_thread'
> NETCDF=/home/users/tadeu33/local/netcdf-3.6.1/lib
> FFTW=/home/users/tadeu33/local/fftw2-2.1.5-1.intel/lib
> MPIDIR=/sw/mpi/intel/openmpi-1.4.2
> MPI_LIBDIR=${MPIDIR}/lib64
> MPI_BINDIR=${MPIDIR}/bin
> MPI_INCLUDEDIR=${MPIDIR}/include
>
> export BLASLAPACK NETCDF FFTW MPIDIR MPI_LIBDIR MPI_BINDIR MPI_INCLUDEDIR
>
> 4.2. Build with fftw from [MKL]
>
> File Makefile from directory src :
>
> INTELLINUX_OPT = -O3 -xssse3
> # LIBS_intellinux=-L${NETCDF}/lib -lnetcdf -L${MKLPATH} -lfftw2xf_intel -
> lmkl_lapack -lmkl_ia32 -lguide -lpthread
> LIBS_intellinux= ${BLASLAPACK} ${NETCDF}/libnetcdf.a
> ${FFTW}/libfftw2xf_intel.a
>
> Environment:
> BLASLAPACK='-L/sw/intel/Compiler/11.1/072/mkl/lib/em64t -lmkl_lapack -
> lmkl_intel_lp64 -lmkl_core -lguide -lpthread -lmkl_intel_thread'
> NETCDF=/home/users/tadeu33/local/netcdf-3.6.1/lib
> FFTW=/home/users/tadeu33/local/fftw2-2.1.5-1.intel/lib
> MPIDIR=/sw/mpi/intel/openmpi-1.4.2
> MPI_LIBDIR=${MPIDIR}/lib64
> MPI_BINDIR=${MPIDIR}/bin
> MPI_INCLUDEDIR=${MPIDIR}/include
>
> export BLASLAPACK NETCDF FFTW MPIDIR MPI_LIBDIR MPI_BINDIR MPI_INCLUDEDIR
>
> Bibliotecas do [MKL] :
> tadeu33 at service0:~/work_20110323/dacapo/src>
> ls /sw/intel/Compiler/11.1/072/mkl/lib/em64t/
> libfftw2xf_intel.a libmkl_blacs_sgimpt_ilp64.a
> libmkl_gf_lp64.a libmkl_intel_thread.a
> libmkl_scalapack_ilp64.a libmkl_vml_avx.so
> libmkl_avx.so libmkl_blacs_sgimpt_lp64.a
> libmkl_gf_lp64.so libmkl_intel_thread.so
> libmkl_scalapack_ilp64.so libmkl_vml_def.so
> libmkl_blacs_ilp64.a libmkl_blas95_ilp64.a
> libmkl_gnu_thread.a libmkl_lapack95_ilp64.a
> libmkl_scalapack_lp64.a libmkl_vml_mc2.so
> libmkl_blacs_intelmpi_ilp64.a libmkl_blas95_lp64.a
> libmkl_gnu_thread.so libmkl_lapack95_lp64.a
> libmkl_scalapack_lp64.so libmkl_vml_mc3.so
> libmkl_blacs_intelmpi_ilp64.so libmkl_cdft_core.a
> libmkl_intel_ilp64.a libmkl_lapack.so
> libmkl_sequential.a libmkl_vml_mc.so
> libmkl_blacs_intelmpi_lp64.a libmkl_core.a
> libmkl_intel_ilp64.so libmkl_mc3.so
> libmkl_sequential.so libmkl_vml_p4n.so
> libmkl_blacs_intelmpi_lp64.so libmkl_core.so
> libmkl_intel_lp64.a libmkl_mc.so
> libmkl_solver_ilp64.a locale
> libmkl_blacs_lp64.a libmkl_def.so
> libmkl_intel_lp64.so libmkl_p4n.so
> libmkl_solver_ilp64_sequential.a
> libmkl_blacs_openmpi_ilp64.a libmkl_gf_ilp64.a
> libmkl_intel_sp2dp.a libmkl_pgi_thread.a libmkl_solver_lp64.a
> libmkl_blacs_openmpi_lp64.a libmkl_gf_ilp64.so
> libmkl_intel_sp2dp.so libmkl_pgi_thread.so
> libmkl_solver_lp64_sequential.a
>
>
> Enviroment Build :
> tadeu33 at service0:~> module list
> Currently Loaded Modulefiles:
> 1) openmpi/1.4.2-intel(default) 3) torque
> 2) gotoblas2/1.13(default) 4) intel/11.1.072
>
> tadeu33 at service0:~> module display openmpi/1.4.2-intel
> -------------------------------------------------------------------
> /sw/modulefiles/openmpi/1.4.2-intel:
>
> module-whatis Sets up Open MPI in your enviornment
> prepend-path PATH /sw/mpi/intel/openmpi-1.4.2/bin
> prepend-path C_INCLUDE_PATH /sw/mpi/intel/openmpi-1.4.2/include
> prepend-path CPLUS_INCLUDE_PATH /sw/mpi/intel/openmpi-1.4.2/include
> prepend-path FPATH /sw/mpi/intel/openmpi-1.4.2/include
> prepend-path LD_LIBRARY_PATH /sw/mpi/intel/openmpi-1.4.2/lib64
> prepend-path LIBRARY_PATH /sw/mpi/intel/openmpi-1.4.2/lib64
> prepend-path MANPATH /sw/mpi/intel/openmpi-1.4.2/share/man
> -------------------------------------------------------------------
>
> tadeu33 at service0:~> module display intel/11.1.072
> -------------------------------------------------------------------
> /sw/modulefiles/intel/11.1.072:
>
> module-whatis Intel Compiler
> module-whatis Category: compiler, runtime support
> module-whatis Description: Intel Compiler Family (C/C++/Fortran for
> x86_64)
> module-whatis URL: http://www.intel.com/cd/software/products/asmo-
> na/eng/compilers/284132.htm
> prepend-path NLSPATH /sw/intel/Compiler/11.1/072/idb/intel64/locale/%l_%
> t/%N
> prepend-path CPATH /sw/intel/Compiler/11.1/072/tbb/include
> prepend-path
> LIBRARY_PATH /sw/intel/Compiler/11.1/072/tbb/em64t/cc4.1.0_libc2.4_kernel2.6.
> 16.21/lib
> prepend-path
> LD_LIBRARY_PATH /sw/intel/Compiler/11.1/072/tbb/em64t/cc4.1.0_libc2.4_kernel2
> .6.16.21/lib
> prepend-path MKLROOT /sw/intel/Compiler/11.1/072/mkl
> prepend-path INCLUDE /sw/intel/Compiler/11.1/072/mkl/include
> prepend-path CPATH /sw/intel/Compiler/11.1/072/mkl/include
> prepend-path FPATH /sw/intel/Compiler/11.1/072/mkl/include
> prepend-path LIBRARY_PATH /sw/intel/Compiler/11.1/072/mkl/lib/em64t
> prepend-path LD_LIBRARY_PATH /sw/intel/Compiler/11.1/072/mkl/lib/em64t
> prepend-path MANPATH /sw/intel/Compiler/11.1/072/mkl/man/en_US
> prepend-path NLSPATH /sw/intel/Compiler/11.1/072/mkl/lib/em64t/locale/%
> l_%t/%N
> prepend-path IPPROOT /sw/intel/Compiler/11.1/072/ipp/em64t
> prepend-path INCLUDE /sw/intel/Compiler/11.1/072/ipp/em64t/include
> prepend-path CPATH /sw/intel/Compiler/11.1/072/ipp/em64t/include
> prepend-path LIB /sw/intel/Compiler/11.1/072/ipp/em64t/lib
> prepend-path LIBRARY_PATH /sw/intel/Compiler/11.1/072/ipp/em64t/lib
> prepend-path
> LD_LIBRARY_PATH /sw/intel/Compiler/11.1/072/ipp/em64t/sharedlib
> prepend-path NLSPATH /sw/intel/Compiler/11.1/072/ipp/em64t/lib/locale/%
> l_%t/%N
> prepend-path PATH /sw/intel/Compiler/11.1/072/bin/intel64
> prepend-path LIBRARY_PATH /sw/intel/Compiler/11.1/072/lib/intel64
> prepend-path LD_LIBRARY_PATH /sw/intel/Compiler/11.1/072/lib/intel64
> prepend-path MANPATH /sw/intel/Compiler/11.1/072/man/en_US
> prepend-path NLSPATH /sw/intel/Compiler/11.1/072/lib/intel64/locale/%l_%
> t/%N
> -------------------------------------------------------------------
>
> Environment that can be add :
>
> tadeu33 at service0:~> module avail
>
> ------------------------------- /sw/modulefiles -----------------------------
> ---
> gotoblas2/1.13(default) mvapich2/1.5.1-p1-gcc
> intel/11.1.064 mvapich2/1.5.1-p1-intel
> intel/11.1.069 netcdf4
> intel/11.1.072(default) openmpi/1.4-gcc
> java/1.6.0_21 openmpi/1.4-intel
> java/1.6.0_23 openmpi/1.4.1-gcc
> java/1.6.0_24(default) openmpi/1.4.1-intel
> maui openmpi/1.4.2-gcc
> mpj/0.35 openmpi/1.4.2-intel(default)
> mpt/2.02(default) openmpi/1.4.3-gcc
> mvapich2/1.4-gcc openmpi/1.4.3-intel
> mvapich2/1.4-intel papi/4.1.0
> mvapich2/1.4.1-gcc papi/4.1.2.1(default)
> mvapich2/1.4.1-intel perfboost
> mvapich2/1.5-gcc torque
> mvapich2/1.5-intel(default)
>
> Netcdf4 environment variables:
> tadeu33 at service0:~> module display netcdf4
> -------------------------------------------------------------------
> /sw/modulefiles/netcdf4:
>
> module-whatis Sets up the NetCDF environment.
> append-path PATH /sw/shared/netcdf/4.0.1/bin
> append-path LIBRARY_PATH /sw/shared/netcdf/4.0.1/lib
> append-path LD_LIBRARY_PATH /sw/shared/netcdf/4.0.1/lib
> setenv NETCDFC_LD -L/sw/shared/netcdf/4.0.1/lib -lnetcdf
> setenv NETCDFF_LD -L/sw/shared/netcdf/4.0.1/lib -lnetcdf
> setenv NETCDFCPP_LD -L/sw/shared/netcdf/4.0.1/lib -lnetcdf_c++
> setenv NETCDF_INC -I/sw/shared/netcdf/4.0.1/include
> setenv NETCDF_INC_F90 -I/sw/shared/netcdf/4.0.1/include
> -------------------------------------------------------------------
>
> Sincerely,
>
> Tadeu Leonardo
>
>
> +++++++++++++++++++++++++++++++++++++++++++++++++++++
>
> PEQ/COPPE renova o nivel de curso 7, maximo na CAPES:
> 45 anos de excelencia no ensino e pesquisa de pos-graduação em
> Engenharia Quimica.
>
> ************************************
>
> PEQ/COPPE : 45 years of commitment to excellence in teaching and
> research in Chemical Engineering.
>
> +++++++++++++++++++++++++++++++++++++++++++++++++++++
>
--
***********************************
Marcin Dulak
Technical University of Denmark
Department of Physics
Building 307, Room 229
DK-2800 Kongens Lyngby
Denmark
Tel.: (+45) 4525 3157
Fax.: (+45) 4593 2399
email: Marcin.Dulak at fysik.dtu.dk
***********************************
More information about the ase-users
mailing list