[gpaw-users] Problem with compiling parallel GPAW

[Michalsky Ronald]

Hi,

I’ve compiled ASE but have difficulties with compiling GPAW for parallel computations (CentOS 6 OS and I’m using a gcc/4.7.2 compiler). Below I describe how I’ve complied Libxc & GPAW. The initial error message (mpicc was not found) is fixed now. However, it appears the parallel executable is not compiled due to some -R option that is not recognized. I’d appreciate your advice.

# Initial email at Campos mailing list:

>>> Assuming Libxc is the only missing dependency:

cd $HOME
wget http://www.tddft.org/programs/octopus/down.php?file=libxc/libxc-2.0.2.tar.gz -O libxc-2.0.2.tar.gz
tar xzf libxc-2.0.2.tar.gz
cd libxc-2.0.2
./configure --enable-shared --prefix=$HOME/xc
make
make install
export C_INCLUDE_PATH=~/xc/include
export LIBRARY_PATH=~/xc/lib
export LD_LIBRARY_PATH=~/xc/lib

cd $HOME
tar xzf gpaw-0.10.0.11364.tar.gz
>>> Rename: $HOME/gpaw-0.10.0.11364 to $HOME/gpaw
export GPAW_HOME =${HOME}/gpaw
cd gpaw

>>> Edit: $HOME/gpaw/customize.py:
libraries = ['acml', 'gfortran']
library_dirs = ['/cluster/home03/mavt/ronaldm/acm15.3.1/gfortran64/lib']
mpicompiler = 'mpicc'
mpilinker = 'gfortran'
mpi_libraries = ['opal_wrapper']

# Reply from From Marcin Dulak

provide full path to mpicc. I see below from the compilation log mpicc 
is not in PATH.

mpicompiler = '/cluster/home03/mavt/ronaldm/openmpi-1.8.1.gfortran/bin/mpicc'
mpilinker = mpicompiler

mpi_libraries = ['opal_wrapper']

no, this is supposed to be a library to be linked (so, for -lmpi it's 
just "mpi"]

mpi_libraries = ['mpi']

> mpi_library_dirs = ['/cluster/home03/mavt/ronaldm/openmpi-1.8.1.gfortran/bin']

'/cluster/home03/mavt/ronaldm/openmpi-1.8.1.gfortran/lib'

# My changes & notes:

>>> I’ve edited in .bashrc:
export PATH=/cluster/home03/mavt/ronaldm/openmpi-1.8.1.gfortran/bin/mpicc:$PATH
>>> I’ve edited in customize.py:
mpicompiler = '/cluster/home03/mavt/ronaldm/openmpi-1.8.1.gfortran/bin/mpicc'
mpilinker = mpicompiler
mpi_libraries = ['mpi']
mpi_library_dirs = ['/cluster/home03/mavt/ronaldm/openmpi-1.8.1.gfortran/lib']
>>> this avoided the error: ‘sh: mpicc: command not found’

# Continue “Initial email at Campos mailing list”:

mpi_library_dirs = ['/cluster/home03/mavt/ronaldm/openmpi-1.8.1.gfortran/bin']
mpi_include_dirs = ['/cluster/home03/mavt/ronaldm/openmpi-1.8.1.gfortran/include']
mpi_runtime_library_dirs = ['/cluster/home03/mavt/ronaldm/openmpi-1.8.1.gfortran/lib']
scalapack = False

>>> Note, $HOME/openmpi-1.8.1.gfortran/bin contains an ‘mpicc’ symlink to the ‘opal_wrapper’ file in the same directory

python setup.py build_ext 2>&1 | tee build_ext.log

>>> yielding:

/usr/bin/gcc -pthread -shared build/temp.linux-x86_64-2.7/c/wigner_seitz.o build/temp.linux-x86_64-2.7/c/mpi.o build/temp.linux-x86_64-2.7/c/spline.o build/temp.linux-x86_64-2.7/c/lcao.o build/temp.linux-x86_64-2.7/c/_gpaw.o build/temp.linux-x86_64-2.7/c/plane_wave.o build/temp.linux-x86_64-2.7/c/symmetry.o build/temp.linux-x86_64-2.7/c/lfc.o build/temp.linux-x86_64-2.7/c/operators.o build/temp.linux-x86_64-2.7/c/lapack.o build/temp.linux-x86_64-2.7/c/point_charges.o build/temp.linux-x86_64-2.7/c/blacs.o build/temp.linux-x86_64-2.7/c/bc.o build/temp.linux-x86_64-2.7/c/transformers.o build/temp.linux-x86_64-2.7/c/lfc2.o build/temp.linux-x86_64-2.7/c/localized_functions.o build/temp.linux-x86_64-2.7/c/mlsqr.o build/temp.linux-x86_64-2.7/c/blas.o build/temp.linux-x86_64-2.7/c/hdf5.o build/temp.linux-x86_64-2.7/c/utilities.o build/temp.linux-x86_64-2.7/c/cerf.o build/temp.linux-x86_64-2.7/c/plt.o build/temp.linux-x86_64-2.7/c/fftw.o build/temp.linux-x86_64-2.7/c/bmgs/bmgs.o build/temp.linux-x86_64-2.7/c/xc/pw91.o build/temp.linux-x86_64-2.7/c/xc/tpss.o build/temp.linux-x86_64-2.7/c/xc/revtpss.o build/temp.linux-x86_64-2.7/c/xc/revtpss_c_pbe.o build/temp.linux-x86_64-2.7/c/xc/pbe.o build/temp.linux-x86_64-2.7/c/xc/libxc.o build/temp.linux-x86_64-2.7/c/xc/rpbe.o build/temp.linux-x86_64-2.7/c/xc/m06l.o build/temp.linux-x86_64-2.7/c/xc/xc_mgga.o build/temp.linux-x86_64-2.7/c/xc/vdw.o build/temp.linux-x86_64-2.7/c/xc/xc.o build/temp.linux-x86_64-2.7/c/xc/ensemble_gga.o -L/cluster/home03/mavt/ronaldm/acm15.3.1/gfortran64/lib -L/cluster/apps/python/2.7.2/x86_64/lib64 -lacml -lgfortran -lpython2.7 -o build/lib.linux-x86_64-2.7/_gpaw.so
sh: mpicc: command not found
gfortran: error: unrecognized option '-R'
mpicc -DNPY_NO_DEPRECATED_API=7 -D_GNU_SOURCE=1 -DPARALLEL=1 -DGPAW_INTERPRETER=1 -Wall -std=c99 -I/cluster/apps/python/2.7.2/x86_64/lib64/python2.7/site-packages/numpy/core/include -I/cluster/home03/mavt/ronaldm/openmpi-1.8.1.gfortran/include -I/cluster/apps/python/2.7.2/x86_64/include/python2.7 -I/cluster/apps/python/2.7.2/x86_64/include/python2.7 -o build/temp.linux-x86_64-2.7/c/bc.o -c c/bc.c
gfortran -o build/bin.linux-x86_64-2.7//gpaw-python build/temp.linux-x86_64-2.7/c/wigner_seitz.o build/temp.linux-x86_64-2.7/c/mpi.o build/temp.linux-x86_64-2.7/c/spline.o build/temp.linux-x86_64-2.7/c/lcao.o build/temp.linux-x86_64-2.7/c/_gpaw.o build/temp.linux-x86_64-2.7/c/plane_wave.o build/temp.linux-x86_64-2.7/c/symmetry.o build/temp.linux-x86_64-2.7/c/lfc.o build/temp.linux-x86_64-2.7/c/operators.o build/temp.linux-x86_64-2.7/c/lapack.o build/temp.linux-x86_64-2.7/c/point_charges.o build/temp.linux-x86_64-2.7/c/blacs.o build/temp.linux-x86_64-2.7/c/bc.o build/temp.linux-x86_64-2.7/c/transformers.o build/temp.linux-x86_64-2.7/c/lfc2.o build/temp.linux-x86_64-2.7/c/localized_functions.o build/temp.linux-x86_64-2.7/c/mlsqr.o build/temp.linux-x86_64-2.7/c/blas.o build/temp.linux-x86_64-2.7/c/hdf5.o build/temp.linux-x86_64-2.7/c/utilities.o build/temp.linux-x86_64-2.7/c/cerf.o build/temp.linux-x86_64-2.7/c/plt.o build/temp.linux-x86_64-2.7/c/fftw.o build/temp.linux-x86_64-2.7/c/bmgs/bmgs.o build/temp.linux-x86_64-2.7/c/xc/pw91.o build/temp.linux-x86_64-2.7/c/xc/tpss.o build/temp.linux-x86_64-2.7/c/xc/revtpss.o build/temp.linux-x86_64-2.7/c/xc/revtpss_c_pbe.o build/temp.linux-x86_64-2.7/c/xc/pbe.o build/temp.linux-x86_64-2.7/c/xc/libxc.o build/temp.linux-x86_64-2.7/c/xc/rpbe.o build/temp.linux-x86_64-2.7/c/xc/m06l.o build/temp.linux-x86_64-2.7/c/xc/xc_mgga.o build/temp.linux-x86_64-2.7/c/xc/vdw.o build/temp.linux-x86_64-2.7/c/xc/xc.o build/temp.linux-x86_64-2.7/c/xc/ensemble_gga.o  -L/cluster/home03/mavt/ronaldm/acm15.3.1/gfortran64/lib -L/cluster/home03/mavt/ronaldm/openmpi-1.8.1.gfortran/bin -L/cluster/apps/python/2.7.2/x86_64/lib64/python2.7/config -lacml -lgfortran -lopal_wrapper -lpython2.7 -lpthread -ldl  -lutil -lm -R/cluster/home03/mavt/ronaldm/openmpi-1.8.1.gfortran/lib  -Xlinker -export-dynamic

* Using standard lapack
* Architecture: linux-x86_64
* Building a custom interpreter
* linking FAILED!  Only serial version of code will work.

# Continue “Reply from From Marcin Dulak”:

> sh: mpicc: command not found
mpicc is missing in PATH.
> gfortran: error: unrecognized option '-R'
i don't see any -R option here, where does it come from?
Please move the thread about building GPAW to the gpaw-users mailing 
list: https://wiki.fysik.dtu.dk/gpaw/mailinglists.html

# My changes & notes:

>>> I’ve written at ‘#Hack taken from distutils to determine option for runtime_libary_dirs’ in $HOME/gpaw/config.py:
    else:
        runtime_lib_option = '-L' # chnaged from '-R' to '-L', assuming this includes linux-x86_64

>>> This yields the error copied below. Note, the error message is the same when not hacking $HOME/gpaw/config.py but instead commenting out in customize.py: 
# mpi_library_dirs = ['/cluster/home03/mavt/ronaldm/openmpi-1.8.1.gfortran/lib']

[…]
/cluster/apps/python/2.7.2/x86_64/lib64/python2.7/config/libpython2.7.a(posixmodule.o): In function `posix_tmpnam':
/tmp/Python-2.7.2/./Modules/posixmodule.c:7370: warning: the use of `tmpnam_r' is dangerous, better use `mkstemp'
/cluster/apps/python/2.7.2/x86_64/lib64/python2.7/config/libpython2.7.a(posixmodule.o): In function `posix_tempnam':
/tmp/Python-2.7.2/./Modules/posixmodule.c:7317: warning: the use of `tempnam' is dangerous, better use `mkstemp'
build/temp.linux-x86_64-2.7/c/xc/libxc.o: In function `lxcXCFunctional_dealloc':
/cluster/home03/mavt/ronaldm/gpaw/c/xc/libxc.c:194: undefined reference to `xc_func_end'
/cluster/home03/mavt/ronaldm/gpaw/c/xc/libxc.c:194: undefined reference to `xc_func_end'
build/temp.linux-x86_64-2.7/c/xc/libxc.o: In function `lxcXCFunctional_CalculateFXC':
/cluster/home03/mavt/ronaldm/gpaw/c/xc/libxc.c:758: undefined reference to `xc_lda_fxc'
/cluster/home03/mavt/ronaldm/gpaw/c/xc/libxc.c:763: undefined reference to `xc_gga_fxc'
/cluster/home03/mavt/ronaldm/gpaw/c/xc/libxc.c:763: undefined reference to `xc_gga_fxc'
/cluster/home03/mavt/ronaldm/gpaw/c/xc/libxc.c:758: undefined reference to `xc_lda_fxc'
build/temp.linux-x86_64-2.7/c/xc/libxc.o: In function `lxcXCFunctional_Calculate':
/cluster/home03/mavt/ronaldm/gpaw/c/xc/libxc.c:642: undefined reference to `xc_lda_exc_vxc'
/cluster/home03/mavt/ronaldm/gpaw/c/xc/libxc.c:647: undefined reference to `xc_gga_exc_vxc'
/cluster/home03/mavt/ronaldm/gpaw/c/xc/libxc.c:647: undefined reference to `xc_gga_exc_vxc'
/cluster/home03/mavt/ronaldm/gpaw/c/xc/libxc.c:653: undefined reference to `xc_mgga_exc_vxc'
/cluster/home03/mavt/ronaldm/gpaw/c/xc/libxc.c:653: undefined reference to `xc_mgga_exc_vxc'
/cluster/home03/mavt/ronaldm/gpaw/c/xc/libxc.c:642: undefined reference to `xc_lda_exc_vxc'
build/temp.linux-x86_64-2.7/c/xc/libxc.o: In function `get_fxc_fd_lda':
/cluster/home03/mavt/ronaldm/gpaw/c/xc/libxc.c:38: undefined reference to `xc_lda_fxc_fd'
build/temp.linux-x86_64-2.7/c/xc/libxc.o: In function `get_point':
/cluster/home03/mavt/ronaldm/gpaw/c/xc/libxc.c:65: undefined reference to `xc_lda_exc_vxc'
/cluster/home03/mavt/ronaldm/gpaw/c/xc/libxc.c:69: undefined reference to `xc_gga_exc_vxc'
build/temp.linux-x86_64-2.7/c/xc/libxc.o: In function `NewlxcXCFunctionalObject':
/cluster/home03/mavt/ronaldm/gpaw/c/xc/libxc.c:852: undefined reference to `xc_family_from_id'
/cluster/home03/mavt/ronaldm/gpaw/c/xc/libxc.c:854: undefined reference to `xc_func_init'
/cluster/home03/mavt/ronaldm/gpaw/c/xc/libxc.c:861: undefined reference to `xc_family_from_id'
/cluster/home03/mavt/ronaldm/gpaw/c/xc/libxc.c:863: undefined reference to `xc_func_init'
/cluster/home03/mavt/ronaldm/gpaw/c/xc/libxc.c:866: undefined reference to `xc_family_from_id'
/cluster/home03/mavt/ronaldm/gpaw/c/xc/libxc.c:868: undefined reference to `xc_func_init'
build/temp.linux-x86_64-2.7/c/xc/libxc.o: In function `lxcXCFuncNum':
/cluster/home03/mavt/ronaldm/gpaw/c/xc/libxc.c:907: undefined reference to `xc_functional_get_number'
build/temp.linux-x86_64-2.7/c/xc/tpss.o: In function `c_tpss_12':
/cluster/home03/mavt/ronaldm/gpaw/c/xc/tpss.c:292: undefined reference to `xc_gga_exc_vxc'
/cluster/home03/mavt/ronaldm/gpaw/c/xc/tpss.c:308: undefined reference to `xc_gga_exc_vxc'
/cluster/home03/mavt/ronaldm/gpaw/c/xc/tpss.c:325: undefined reference to `xc_gga_exc_vxc'
/cluster/home03/mavt/ronaldm/gpaw/c/xc/tpss.c:292: undefined reference to `xc_gga_exc_vxc'
/cluster/home03/mavt/ronaldm/gpaw/c/xc/tpss.c:308: undefined reference to `xc_gga_exc_vxc'
build/temp.linux-x86_64-2.7/c/xc/tpss.o:/cluster/home03/mavt/ronaldm/gpaw/c/xc/tpss.c:325: more undefined references to `xc_gga_exc_vxc' follow
build/temp.linux-x86_64-2.7/c/xc/tpss.o: In function `x_tpss_para':
/cluster/home03/mavt/ronaldm/gpaw/c/xc/tpss.c:125: undefined reference to `xc_lda_exc_vxc'
build/temp.linux-x86_64-2.7/c/xc/tpss.o: In function `tpss_end':
/cluster/home03/mavt/ronaldm/gpaw/c/xc/tpss.c:544: undefined reference to `xc_func_end'
/cluster/home03/mavt/ronaldm/gpaw/c/xc/tpss.c:547: undefined reference to `xc_func_end'
/cluster/home03/mavt/ronaldm/gpaw/c/xc/tpss.c:548: undefined reference to `xc_func_end'
build/temp.linux-x86_64-2.7/c/xc/tpss.o: In function `tpss_init':
/cluster/home03/mavt/ronaldm/gpaw/c/xc/tpss.c:534: undefined reference to `xc_func_init'
/cluster/home03/mavt/ronaldm/gpaw/c/xc/tpss.c:538: undefined reference to `xc_func_init'
/cluster/home03/mavt/ronaldm/gpaw/c/xc/tpss.c:539: undefined reference to `xc_func_init'
build/temp.linux-x86_64-2.7/c/xc/revtpss.o: In function `revtpss_end':
/cluster/home03/mavt/ronaldm/gpaw/c/xc/revtpss.c:553: undefined reference to `xc_func_end'
/cluster/home03/mavt/ronaldm/gpaw/c/xc/revtpss.c:556: undefined reference to `xc_func_end'
build/temp.linux-x86_64-2.7/c/xc/revtpss.o: In function `revtpss_init':
/cluster/home03/mavt/ronaldm/gpaw/c/xc/revtpss.c:545: undefined reference to `xc_func_init'
/cluster/home03/mavt/ronaldm/gpaw/c/xc/revtpss.c:547: undefined reference to `xc_func_init'
build/temp.linux-x86_64-2.7/c/xc/revtpss.o: In function `x_revtpss_para':
/cluster/home03/mavt/ronaldm/gpaw/c/xc/revtpss.c:449: undefined reference to `xc_lda_exc_vxc'
build/temp.linux-x86_64-2.7/c/xc/revtpss.o: In function `revtpss_end':
/cluster/home03/mavt/ronaldm/gpaw/c/xc/revtpss.c:557: undefined reference to `xc_func_end'
build/temp.linux-x86_64-2.7/c/xc/revtpss.o: In function `revtpss_init':
/cluster/home03/mavt/ronaldm/gpaw/c/xc/revtpss.c:548: undefined reference to `xc_func_init'
build/temp.linux-x86_64-2.7/c/xc/revtpss_c_pbe.o: In function `xc_perdew_params':
/cluster/home03/mavt/ronaldm/gpaw/c/xc/revtpss_c_pbe.c:64: undefined reference to `xc_lda_exc'
/cluster/home03/mavt/ronaldm/gpaw/c/xc/revtpss_c_pbe.c:70: undefined reference to `xc_lda'
/cluster/home03/mavt/ronaldm/gpaw/c/xc/revtpss_c_pbe.c:67: undefined reference to `xc_lda_exc_vxc'
build/temp.linux-x86_64-2.7/c/xc/m06l.o: In function `m06l_end':
/cluster/home03/mavt/ronaldm/gpaw/c/xc/m06l.c:733: undefined reference to `xc_func_end'
/cluster/home03/mavt/ronaldm/gpaw/c/xc/m06l.c:736: undefined reference to `xc_func_end'
build/temp.linux-x86_64-2.7/c/xc/m06l.o: In function `m06l_init':
/cluster/home03/mavt/ronaldm/gpaw/c/xc/m06l.c:724: undefined reference to `xc_func_init'
build/temp.linux-x86_64-2.7/c/xc/m06l.o: In function `c_m06l_para':
/cluster/home03/mavt/ronaldm/gpaw/c/xc/m06l.c:328: undefined reference to `xc_lda_exc_vxc'
/cluster/home03/mavt/ronaldm/gpaw/c/xc/m06l.c:338: undefined reference to `xc_lda_exc_vxc'
/cluster/home03/mavt/ronaldm/gpaw/c/xc/m06l.c:372: undefined reference to `xc_lda_exc_vxc'
build/temp.linux-x86_64-2.7/c/xc/m06l.o: In function `x_m06l_para':
/cluster/home03/mavt/ronaldm/gpaw/c/xc/m06l.c:656: undefined reference to `xc_gga_exc_vxc'
build/temp.linux-x86_64-2.7/c/xc/m06l.o: In function `m06l_init':
/cluster/home03/mavt/ronaldm/gpaw/c/xc/m06l.c:727: undefined reference to `xc_func_init'
collect2: error: ld returned 1 exit status
python2.7 -I/cluster/apps/python/2.7.2/x86_64/include/python2.7 -o build/temp.linux-x86_64-2.7/c/hdf5.o -c c/hdf5.c
/cluster/home03/mavt/ronaldm/openmpi-1.8.1.gfortran/bin/mpicc -o build/bin.linux-x86_64-2.7//gpaw-python build/temp.linux-x86_64-2.7/c/mlsqr.o build/temp.linux-x86_64-2.7/c/symmetry.o build/temp.linux-x86_64-2.7/c/lapack.o build/temp.linux-x86_64-2.7/c/lcao.o build/temp.linux-x86_64-2.7/c/blacs.o build/temp.linux-x86_64-2.7/c/plt.o build/temp.linux-x86_64-2.7/c/plane_wave.o build/temp.linux-x86_64-2.7/c/wigner_seitz.o build/temp.linux-x86_64-2.7/c/utilities.o build/temp.linux-x86_64-2.7/c/spline.o build/temp.linux-x86_64-2.7/c/point_charges.o build/temp.linux-x86_64-2.7/c/_gpaw.o build/temp.linux-x86_64-2.7/c/mpi.o build/temp.linux-x86_64-2.7/c/lfc2.o build/temp.linux-x86_64-2.7/c/bc.o build/temp.linux-x86_64-2.7/c/hdf5.o build/temp.linux-x86_64-2.7/c/cerf.o build/temp.linux-x86_64-2.7/c/transformers.o build/temp.linux-x86_64-2.7/c/blas.o build/temp.linux-x86_64-2.7/c/lfc.o build/temp.linux-x86_64-2.7/c/localized_functions.o build/temp.linux-x86_64-2.7/c/operators.o build/temp.linux-x86_64-2.7/c/fftw.o build/temp.linux-x86_64-2.7/c/bmgs/bmgs.o build/temp.linux-x86_64-2.7/c/xc/libxc.o build/temp.linux-x86_64-2.7/c/xc/xc_mgga.o build/temp.linux-x86_64-2.7/c/xc/tpss.o build/temp.linux-x86_64-2.7/c/xc/pw91.o build/temp.linux-x86_64-2.7/c/xc/pbe.o build/temp.linux-x86_64-2.7/c/xc/revtpss.o build/temp.linux-x86_64-2.7/c/xc/xc.o build/temp.linux-x86_64-2.7/c/xc/revtpss_c_pbe.o build/temp.linux-x86_64-2.7/c/xc/ensemble_gga.o build/temp.linux-x86_64-2.7/c/xc/vdw.o build/temp.linux-x86_64-2.7/c/xc/rpbe.o build/temp.linux-x86_64-2.7/c/xc/m06l.o  -L/cluster/home03/mavt/ronaldm/acm15.3.1/gfortran64/lib -L/cluster/home03/mavt/ronaldm/openmpi-1.8.1.gfortran/lib -L/cluster/apps/python/2.7.2/x86_64/lib64/python2.7/config -lacml -lgfortran -lmpi -lpython2.7 -lpthread -ldl  -lutil -lm -L/cluster/home03/mavt/ronaldm/openmpi-1.8.1.gfortran/lib  -Xlinker -export-dynamic

* Using standard lapack
* Architecture: linux-x86_64
* Building a custom interpreter
* linking FAILED!  Only serial version of code will work.

Ronald Michalsky
Postdoctoral Research Associate
ETH Zürich
Institute of Energy Technology
ML K 23, Sonneggstr. 3
8092 Zürich, Switzerland
Tel: +41-44-6338383