[gpaw-users] Library-linking issue while compiling gpaw using mkl library

Thu Jul 15 00:04:50 CEST 2010

Hi Kamal,

I'm not sure exactly what might cause that error. What happens if you run the automatic test suite on multiple processors (for example, [username]$ gpaw-test -j 4 )? This might let you determine if the problem is limited to SCALAPACK/BLACS or is a broader problem that occurs with other kinds of MPI communication.

Mike

On Jul 14, 2010, at 5:09 PM, ksaha at physics.udel.edu wrote:

> Dear Mike,
> 
> Thank you for your quick reply. Using your customize.py and within Openmpi,
> some of the things work (like H.py). However I get a nasty error when I
> try to run CH4.py. Here is the error message:
> 
> /usr/lib64/python2.4/site-packages/scipy/misc/__init__.py:25:
> DeprecationWarning: NumpyTest will be removed in the next release; please
> update your code to use nose or unittest
>  test = NumpyTest().test
> /usr/lib64/python2.4/site-packages/scipy/misc/__init__.py:25:
> DeprecationWarning: NumpyTest will be removed in the next release; please
> update your code to use nose or unittest
>  test = NumpyTest().test
> [compute-0-1:25230] *** Process received signal ***
> [compute-0-1:25230] Signal: Segmentation fault (11)
> [compute-0-1:25230] Signal code: Address not mapped (1)
> [compute-0-1:25230] Failing at address: 0x44000098
> [compute-0-11:13731] *** Process received signal ***
> [compute-0-11:13731] Signal: Segmentation fault (11)
> [compute-0-11:13731] Signal code: Address not mapped (1)
> [compute-0-11:13731] Failing at address: 0x44000098
> [compute-0-1:25230] [ 0] /lib64/libpthread.so.0 [0x38f620e4c0]
> [compute-0-1:25230] [ 1]
> /share/apps/openmpi/intel//lib/libmpi.so.0(MPI_Comm_size+0x60)
> [0x2b66d5cd1080]
> [compute-0-1:25230] [ 2]
> /share/apps/intel/mkl/10.2.0.013/lib/em64t/libmkl_blacs_intelmpi_lp64.so(Cblacs_pinfo+0xa4)
> [0x2b66d4e43864]
> [compute-0-1:25230] *** End of error message ***
> [compute-0-11:13731] [ 0] /lib64/libpthread.so.0 [0x3c7880e4c0]
> [compute-0-11:13731] [ 1]
> /share/apps/openmpi/intel//lib/libmpi.so.0(MPI_Comm_size+0x60)
> [0x2b8eae8ab080]
> [compute-0-11:13731] [ 2]
> /share/apps/intel/mkl/10.2.0.013/lib/em64t/libmkl_blacs_intelmpi_lp64.so(Cblacs_pinfo+0xa4)
> [0x2b8eada1d864]
> [compute-0-11:13731] *** End of error message ***
> --------------------------------------------------------------------------
> mpirun noticed that process rank 0 with PID 25230 on node
> compute-0-1.local exited on signal 11 (Segmentation fault).
> --------------------------------------------------------------------------
> 
> Do you have any suggestions to resolve this? Thank you!
> 
> Regards,
> Kamal
> 
> 
> 
> 
>> Dear Kamal,
>> 
>> This customize.py works for me:
>> 
>> scalapack = True
>> 
>> compiler = 'mpicc -O3 -xHOST -ipo -static -no-prec-div -std=c99 -fPIC'
>> 
>> mpicompiler = 'mpicc -O3 -xHOST -ipo -static -no-prec-div -std=c99 -fPIC'
>> 
>> library_dirs += ['/home2/mrw47/intel/mkl/10.2.3.029/lib/em64t/']
>> 
>> libraries = ['mkl_scalapack_lp64',
>>             'mkl_blacs_openmpi_lp64',
>>             'mkl_lapack',
>>             'mkl_intel_lp64',
>>             'mkl_sequential',
>>             'mkl_core',
>>             'iomp5',
>>             'pthread']
>> 
>> define_macros += [('GPAW_NO_UNDERSCORE_CBLACS', '1')]
>> define_macros += [('GPAW_NO_UNDERSCORE_CSCALAPACK', '1')]
>> 
>> You can just replace 'mkl_blacs_openmpi_lp64' with
>> 'mkl_blacs_intelmpi_lp64' if you are set up to use Intel's MPI instead of
>> OpenMPI. Perhaps it is worth checking that the first 'mpicc' that is found
>> is actually linked to the Intel MPI libraries and not OpenMPI ones. I'm
>> not an MPI expert, but my impression is that OpenMPI is much more
>> standard. Hope that helps,
>> 
>> Mike
>> 
>> On Jul 13, 2010, at 4:19 PM, ksaha at physics.udel.edu wrote:
>> 
>>> Dear gpaw-users,
>>> 
>>> Did anybody get success in compliling the gpaw using intel mkl library?
>>> I
>>> am getting some linking issue as follows:
>>> 
>>> mpicc -o build/bin.linux-x86_64-2.4//gpaw-python
>>> build/temp.linux-x86_64-2.4/c/xc.o build/temp.linux-x86_64-2.4/c/blas.o
>>> build/temp.linux-x86_64-2.4/c/rpbe.o
>>> build/temp.linux-x86_64-2.4/c/lfc2.o
>>> build/temp.linux-x86_64-2.4/c/compiled_WITH_SL.o
>>> build/temp.linux-x86_64-2.4/c/mgga.o
>>> build/temp.linux-x86_64-2.4/c/pw91.o
>>> build/temp.linux-x86_64-2.4/c/vasiliev02prb.o
>>> build/temp.linux-x86_64-2.4/c/lapack.o
>>> build/temp.linux-x86_64-2.4/c/plane_wave.o
>>> build/temp.linux-x86_64-2.4/c/libxc.o
>>> build/temp.linux-x86_64-2.4/c/spline.o
>>> build/temp.linux-x86_64-2.4/c/utilities.o
>>> build/temp.linux-x86_64-2.4/c/tpss_ec.o
>>> build/temp.linux-x86_64-2.4/c/mpi.o build/temp.linux-x86_64-2.4/c/vdw.o
>>> build/temp.linux-x86_64-2.4/c/wigner_seitz.o
>>> build/temp.linux-x86_64-2.4/c/_gpaw.o
>>> build/temp.linux-x86_64-2.4/c/mlsqr.o
>>> build/temp.linux-x86_64-2.4/c/hdf5.o
>>> build/temp.linux-x86_64-2.4/c/ensemble_gga.o
>>> build/temp.linux-x86_64-2.4/c/transformers.o
>>> build/temp.linux-x86_64-2.4/c/localized_functions.o
>>> build/temp.linux-x86_64-2.4/c/lcao.o
>>> build/temp.linux-x86_64-2.4/c/lb94.o
>>> build/temp.linux-x86_64-2.4/c/pbe.o
>>> build/temp.linux-x86_64-2.4/c/blacs.o
>>> build/temp.linux-x86_64-2.4/c/symmetry.o
>>> build/temp.linux-x86_64-2.4/c/tpss.o
>>> build/temp.linux-x86_64-2.4/c/point_charges.o
>>> build/temp.linux-x86_64-2.4/c/tpss_ex.o
>>> build/temp.linux-x86_64-2.4/c/d2Excdn2.o
>>> build/temp.linux-x86_64-2.4/c/cerf.o build/temp.linux-x86_64-2.4/c/bc.o
>>> build/temp.linux-x86_64-2.4/c/lfc.o build/temp.linux-x86_64-2.4/c/plt.o
>>> build/temp.linux-x86_64-2.4/c/operators.o
>>> build/temp.linux-x86_64-2.4/c/io_wrappers.o
>>> build/temp.linux-x86_64-2.4/c/bmgs/bmgs.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/util.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/special_functions.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/gga_x_am05.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/mgga.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/gga_perdew.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/hyb_gga.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/lda_c_pz.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/gga_xc_hcth.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/gga_c_am05.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/functionals.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/hyb_gga_xc_b3lyp.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/mgga_c_tpss.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/mgga_x_m06l.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/lda_c_xalpha.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/gga_x_pbe.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/hyb_gga_xc_b3p86.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/gga_x_b86.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/gga_x_b88.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/gga_x_dk87.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/hyb_gga_xc_b3pw91.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/mgga_c_m06l.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/gga_x_lg93.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/lda_xc_teter93.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/gga_x_optx.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/gga_x_pbea.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/gga_x_pw86.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/lda_c_rpa.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/lda_x.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/lca.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/hyb_gga_xc_b1wc.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/gga_x_b86_mgc.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/lda_c_wigner.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/gga_c_pw91.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/gga_x_wc.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/lda_c_pw.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/gga_c_lyp.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/hyb_gga_xc_x3lyp.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/gga_xc_lb.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/mgga_x_tpss.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/gga_x_g96.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/gga_x_ft97.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/gga_x_pw91.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/gga_x_mpbe.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/mgga_c_revtpss.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/lda_c_vwn.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/gga_xc_edf1.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/hyb_gga_xc_o3lyp.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/mgga_x_revtpss.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/gga_xc_xlyp.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/lda.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/gga.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/lda_c_amgb.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/gga_c_pbe.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/lca_omc.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/hyb_gga_xc_pbeh.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/gga_x_rpbe.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/lda_c_hl.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/gga_c_p86.o
>>> build/temp.linux-x86_64-2.4/c/libxc/src/lca_lch.o
>>> -L/usr/lib64/python2.4/config -lblas -llapack -lpython2.4 -lpthread -ldl
>>> -lutil -lm  -L/share/apps/intel/mkl/10.2.0.013/lib/em64t
>>> /share/apps/intel/mkl/10.2.0.013/lib/em64t/libmkl_scalapack_lp64.a
>>> /share/apps/intel/mkl/10.2.0.013/lib/em64t/libmkl_solver_lp64_sequential.a
>>> -Wl,--start-group
>>> /share/apps/intel/mkl/10.2.0.013/lib/em64t/libmkl_intel_lp64.a
>>> /share/apps/intel/mkl/10.2.0.013/lib/em64t/libmkl_sequential.a
>>> /share/apps/intel/mkl/10.2.0.013/lib/em64t/libmkl_core.a
>>> /share/apps/intel/mkl/10.2.0.013/lib/em64t/libmkl_blacs_intelmpi_lp64.a
>>> -Wl,--end-group -lpthread -L/usr/kerberos/lib64 -Xlinker -export-dynamic
>>> /usr/lib64/python2.4/config/libpython2.4.a(posixmodule.o): In function
>>> `posix_tmpnam':
>>> (.text+0x665): warning: the use of `tmpnam_r' is dangerous, better use
>>> `mkstemp'
>>> /usr/lib64/python2.4/config/libpython2.4.a(posixmodule.o): In function
>>> `posix_tempnam':
>>> (.text+0x748): warning: the use of `tempnam' is dangerous, better use
>>> `mkstemp'
>>> build/temp.linux-x86_64-2.4/c/lapack.o: warning: memset used with
>>> constant
>>> zero length parameter; this could be due to transposed parameters
>>> build/temp.linux-x86_64-2.4/c/mpi.o: In function `diagonalize':
>>> c/mpi.c:(.text+0x33bb): undefined reference to `Cblacs_gridinit_'
>>> c/mpi.c:(.text+0x33fa): undefined reference to `Cblacs_gridinfo_'
>>> c/mpi.c:(.text+0x345b): undefined reference to `Cblacs_pnum_'
>>> c/mpi.c:(.text+0x4060): undefined reference to `Cpdgemr2d_'
>>> c/mpi.c:(.text+0x40d3): undefined reference to `Cblacs_gridexit_'
>>> build/temp.linux-x86_64-2.4/c/mpi.o: In function `inverse_cholesky':
>>> c/mpi.c:(.text+0x4379): undefined reference to `Cblacs_gridinit_'
>>> c/mpi.c:(.text+0x43b8): undefined reference to `Cblacs_gridinfo_'
>>> c/mpi.c:(.text+0x4419): undefined reference to `Cblacs_pnum_'
>>> c/mpi.c:(.text+0x488c): undefined reference to `Cpdgemr2d_'
>>> c/mpi.c:(.text+0x4943): undefined reference to `Cblacs_gridexit_'
>>> build/temp.linux-x86_64-2.4/c/blacs.o: In function `new_blacs_context':
>>> c/blacs.c:(.text+0x819): undefined reference to `Cblacs_pinfo_'
>>> c/blacs.c:(.text+0x84b): undefined reference to `Cblacs_gridinit_'
>>> build/temp.linux-x86_64-2.4/c/blacs.o: In function `get_blacs_gridinfo':
>>> c/blacs.c:(.text+0x8b5): undefined reference to `Cblacs_gridinfo_'
>>> build/temp.linux-x86_64-2.4/c/blacs.o: In function
>>> `get_blacs_local_shape':
>>> c/blacs.c:(.text+0x94a): undefined reference to `Cblacs_gridinfo_'
>>> build/temp.linux-x86_64-2.4/c/blacs.o: In function `blacs_destroy':
>>> c/blacs.c:(.text+0x9c0): undefined reference to `Cblacs_gridexit_'
>>> build/temp.linux-x86_64-2.4/c/blacs.o: In function `scalapack_redist':
>>> c/blacs.c:(.text+0xbbb): undefined reference to `Cpdtrmr2d_'
>>> c/blacs.c:(.text+0xc25): undefined reference to `Cpztrmr2d_'
>>> c/blacs.c:(.text+0xc9a): undefined reference to `Cpdgemr2d_'
>>> c/blacs.c:(.text+0xcf3): undefined reference to `Cpzgemr2d_'
>>> build/temp.linux-x86_64-2.4/c/blacs.o: In function
>>> `scalapack_diagonalize_ex':
>>> c/blacs.c:(.text+0x128d): undefined reference to `Cblacs_gridinfo_'
>>> build/temp.linux-x86_64-2.4/c/blacs.o: In function
>>> `scalapack_general_diagonalize_ex':
>>> c/blacs.c:(.text+0x2495): undefined reference to `Cblacs_gridinfo_'
>>> 
>>> * Using standard lapack
>>> * Compiling gpaw with mpiicc
>>> * Compiling with ScaLapack
>>> * Architecture: linux-x86_64
>>> * Building a custom interpreter
>>> * linking FAILED!  Only serial version of code will work.
>>> 
>>> Here is the mkl library I put in customize.py:
>>> 
>>> extra_link_args = ['-L/share/apps/intel/mkl/10.2.0.013/lib/em64t
>>> /share/apps/intel/mkl/10.2.0.013/lib/em64t/libmkl_scalapack_lp64.a
>>> /share/apps/intel/mkl/10.2.0.013/lib/em64t/libmkl_solver_lp64_sequential.a
>>> -Wl,--start-group
>>> /share/apps/intel/mkl/10.2.0.013/lib/em64t/libmkl_intel_lp64.a
>>> /share/apps/intel/mkl/10.2.0.013/lib/em64t/libmkl_sequential.a
>>> /share/apps/intel/mkl/10.2.0.013/lib/em64t/libmkl_core.a
>>> /share/apps/intel/mkl/10.2.0.013/lib/em64t/libmkl_blacs_intelmpi_lp64.a
>>> -Wl,--end-group -lpthread']
>>> 
>>> Thanks in advance for your suggestion!
>>> 
>>> Regards.
>>> Kamal
>>> 
>>> ==========================================================
>>> Kamal K. Saha
>>> Department of Physics and Astronomy
>>> University of Delaware
>>> Newark, DE 19716-2570
>>> Phone: (302) 831-8519, Fax: (302) 831-1637
>>> 
>>> http://web.physics.udel.edu/about/directory/postdoc/kamal-k-saha
>>> ==========================================================
>>> 
>>> _______________________________________________
>>> gpaw-users mailing list
>>> gpaw-users at listserv.fysik.dtu.dk
>>> https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users
>> 
>> 
> 
> 
> ==========================================================
> Dr. Kamal K. Saha
> Department of Physics and Astronomy
> University of Delaware
> Newark, DE 19716-2570
> Phone: (302) 831-8519, Fax: (302) 831-1637
> 
> http://web.physics.udel.edu/about/directory/postdoc/kamal-k-saha
> ==========================================================