[gpaw-users] Trouble Compiling GPAW
Ask Hjorth Larsen
asklarsen at gmail.com
Fri Mar 18 11:52:27 CET 2016
Hello
2016-03-17 19:57 GMT+01:00 Tomlinson, Warren (CDR) <wwtomlin at nps.edu>:
> All-
> Please ignore my installation problem form my previous email (below). Problem has been solved.
>
> On an unrelated note I have a question about a large system I’m working with (656 atoms). It is the unit cell for the UiO-67 MOF (periodic in all three directions). I’m trying to relax the structure and I’ve gotten to a point where the energy is not changing much at all between SCF steps, but the relaxation keeps running. The system seemed to settle relatively quickly to -4163.05 eV (about 200 or so SCF steps), but since then there have been 1063 SCF steps and the energy went from -4163.026398 eV to -4163.069115 eV. The lowest energy over those 1063 SCF steps was -4163.074407. It seems to me that the system is just shifting around a minimum, but never quite settling there. Do you have any tips for addressing this?
>
> Below is my python file:
>
> from ase import Atoms
> from ase.io import read, write
> from ase.ga.relax_attaches import VariansBreak
> from gpaw import GPAW, Mixer, FermiDirac
> from ase.optimize import QuasiNewton
> from gpaw.poisson import PoissonSolver
> import numpy as np
>
> cell = read('Full_N_Pd_szp_ptl.pdb')
> cell.set_cell([26.52,26.52,26.52])
> cell.set_pbc(1)
>
> calc_LCAO = GPAW(mode='lcao',
> gpts=(144,144,144),
> xc='PBE',
> poissonsolver=PoissonSolver(relax='GS', eps=1e-10),
> parallel={'band':2,'sl_default':(24,24,64)},
> basis = 'szp',
> mixer=Mixer(0.1, 5, weight=100.0),
Try lowering the mixer parameter from 0.1 to 0.05. Larger systems
tend to require more conservative mixing.
If it does not work, please attach the logfile.
It looks like you are using a very large number of cores (576). On
normal architectures this should probably run on 24-48 cores, or even
less given that it is only szp. Adding more cores beyond that may
make it faster, but it is very unlikely that it strong-scales well to
576.
Best regards
Ask
> occupations=FermiDirac(width=0.1),
> txt='Full_N_Pd_szp.out',
> maxiter=1000
> )
>
> cell.set_calculator(calc_LCAO)
> opt = QuasiNewton(cell)
> vb = VariansBreak(cell, opt)
> opt.attach(vb.write)
> opt.run()
>
>
> Thanks,
> Warren
>
>
>> On Mar 15, 2016, at 12:16 PM, Tomlinson, Warren (CDR) <wwtomlin at nps.edu> wrote:
>>
>> Hello-
>> I’m having an issue compiling GPAW on a cluster and I’m wondering if someone can help.
>>
>> The system is a Cray XC30 cluster (called Lightning)
>> I am attempting to compile in the GNU environment (although I get the same error when trying with Intel)
>>
>> Relevant modules:
>> gcc/5.1.0
>> cray-mpich/7.2.6
>> cray-libsci/13.2.0
>> python/gnu/2.7.9
>> numpy/gnu/1.9.2
>> spicy/gnu/0.15.1
>>
>>
>> Relevant excerpts from system user’s manual:
>> 5.1.1. Message Passing Interface (MPI)
>>
>> This release of MPI-2 derives from Argonne National Laboratory MPICH-2 and implements the MPI-2.2 standard except for spawn support, as documented by the MPI Forum in "MPI: A Message Passing Interface Standard, Version 2.2."
>>
>> The Message Passing Interface (MPI) is part of the software support for parallel programming across a network of computer systems through a technique known as message passing. MPI establishes a practical, portable, efficient, and flexible standard for message passing that makes use of the most attractive features of a number of existing message-passing systems, rather than selecting one of them and adopting it as the standard. See "man intro_mpi" for additional information.
>>
>> When creating an MPI program on Lightning, ensure the following:
>>
>> • That the default MPI module (cray-mpich) has been loaded. To check this, run the "module list" command. If cray-mpich is not listed and a different MPI module is listed, use the following command:
>> module swap other_mpi_module cray-mpich
>>
>> If no MPI module is loaded, load the cray-mpich module.
>>
>> module load cray-mpich
>>
>> • That the source code includes one of the following lines:
>> INCLUDE "mpif.h" ## for Fortran, or
>> #include <mpi.h> ## for C/C++
>>
>> To compile an MPI program, use the following examples:
>>
>> ftn -o pi_program mpi_program.f
>> ## for Fortran, or
>> cc -o mpi_program mpi_program.c ## for C/C++
>>
>> ———————————————————————
>>
>> 5.1.3. Open Multi-Processing (OpenMP)
>>
>> OpenMP is a portable, scalable model that gives programmers a simple and flexible interface for developing parallel applications. It supports shared-memory multiprocessing programming in C, C++ and Fortran, and consists of a set of compiler directives, library routines, and environment variables that influence compilation and run-time behavior.
>>
>> When creating an OpenMP program on Lightning, ensure the following:
>>
>> • That the default MPI module (cray-mpich) has been loaded. To check this, run the "module list" command. If cray-mpich is not listed and a different MPI module is listed, use the following command:
>> module swap other_mpi_module cray-mpich
>>
>> If no MPI module is loaded, load the cray-mpich module.
>>
>> module load cray-mpich
>>
>> • That if using OpenMP functions (for example, omp_get_wtime), the source code includes one of the following lines:
>>
>> INCLUDE 'omp.h' ## for Fortran, or
>> #include <omp.h> ## for C/C++
>>
>> Or, if the code is written in Fortran 90 or later, the following line may be used instead:
>>
>> USE omp_lib
>>
>> • That the compile command includes an option to reference the OpenMP library. The PGI, Cray, Intel, and GNU compilers support OpenMP, and each one uses a different option.
>>
>> To compile an OpenMP program, use the following examples:
>>
>> For C/C++ codes:
>>
>> cc -o OpenMP_program -mp=nonuma OpenMP_program.c ## PGI
>> cc -o OpenMP_program -h omp OpenMP_program.c ## Cray
>> cc -o OpenMP_program -openmp OpenMP_program.c ## Intel
>> cc -o OpenMP_program -fopenmp OpenMP_program.c ## GNU
>>
>> ———————————————————————
>>
>> 5.2. Available Compilers
>>
>> Lightning has four programming environment suites.
>>
>> • Portland Group (PGI)
>> • Cray Fortran and C/C++
>> • Intel
>> • GNU
>> On Lightning, different sets of compilers are used to compile codes for serial vs. parallel execution.
>>
>> Compiling for the Compute Nodes
>>
>> Codes compiled to run on the compute nodes may be serial or parallel. The x86-64 instruction set for Intel Ivy Bridge E5-2697 processors has extensions for the Floating Point Unit (FPU) that require the module craype-ivybridge to be loaded. This module is loaded for you by default. To compile codes for execution on the compute nodes, the same compile commands are available in all programming environment suites as shown in the following table:
>>
>> Compute Node Compiler Commands
>> Language PGI
>> Cray Intel
>> GNU Serial/Parallel
>> C cc
>> cc cc
>> cc Serial/Parallel
>> C++ CC
>> CC CC
>> CC Serial/Parallel
>> Fortran 77 f77
>> f77 f77
>> f77 Serial/Parallel
>> Fortran 90 ftn
>> ftn ftn
>> ftn Serial/Parallel
>> ——————————————————————————————
>>
>>
>> Building the serial version of GPAW gives me no trouble. I have compiled it and run all the tests and they all pass. When trying to build the custom interpreter, the compiling goes fine, but the linking fails.
>>
>> Below is the link line causing the issue:
>> cc -o build/bin.linux-x86_64-2.7//gpaw-python build/temp.linux-x86_64-2.7/c/woperators.o build/temp.linux-x86_64-2.7/c/plt.o build/temp.linux-x86_64-2.7/c/lapack.o build/temp.linux-x86_64-2.7/c/symmetry.o build/temp.linux-x86_64-2.7/c/plane_wave.o build/temp.linux-x86_64-2.7/c/operators.o build/temp.linux-x86_64-2.7/c/mlsqr.o build/temp.linux-x86_64-2.7/c/transformers.o build/temp.linux-x86_64-2.7/c/utilities.o build/temp.linux-x86_64-2.7/c/spline.o build/temp.linux-x86_64-2.7/c/lfc2.o build/temp.linux-x86_64-2.7/c/localized_functions.o build/temp.linux-x86_64-2.7/c/wigner_seitz.o build/temp.linux-x86_64-2.7/c/mpi.o build/temp.linux-x86_64-2.7/c/lfc.o build/temp.linux-x86_64-2.7/c/bc.o build/temp.linux-x86_64-2.7/c/hdf5.o build/temp.linux-x86_64-2.7/c/blas.o build/temp.linux-x86_64-2.7/c/fftw.o build/temp.linux-x86_64-2.7/c/lcao.o build/temp.linux-x86_64-2.7/c/point_charges.o build/temp.linux-x86_64-2.7/c/_gpaw.o build/temp.linux-x86_64-2.7/c/cerf.o build/temp.linux-x86_64-2.7/c/blacs.o build/temp.linux-x86_64-2.7/c/bmgs/bmgs.o build/temp.linux-x86_64-2.7/c/xc/rpbe.o build/temp.linux-x86_64-2.7/c/xc/tpss.o build/temp.linux-x86_64-2.7/c/xc/xc.o build/temp.linux-x86_64-2.7/c/xc/revtpss_c_pbe.o build/temp.linux-x86_64-2.7/c/xc/pbe.o build/temp.linux-x86_64-2.7/c/xc/libxc.o build/temp.linux-x86_64-2.7/c/xc/m06l.o build/temp.linux-x86_64-2.7/c/xc/pw91.o build/temp.linux-x86_64-2.7/c/xc/revtpss.o build/temp.linux-x86_64-2.7/c/xc/ensemble_gga.o build/temp.linux-x86_64-2.7/c/xc/xc_mgga.o build/temp.linux-x86_64-2.7/c/xc/vdw.o -L/home/wwtomlin/xc/lib -L/opt/gcc/5.1.0/snos/lib64 -L/opt/cray/libsci/13.2.0/GNU/5.1/x86_64/lib -L/app/COST/python/2.7.9/gnu/lib -L/opt/cray/mpt/7.2.6/gni/mpich-gnu/51/lib -L/app/COST/python/2.7.9/gnu/lib/python2.7/config -lxc -lpython2.7 -lpthread -ldl -lutil -lm -pg -L. -L/app/COST/bzip2/1.0.6/gnu//lib -L/app/COST/tcltk/8.6.4/gnu//lib -L/app/COST/dependencies/sqlite/3081101/gnu//lib -L/app/COST/dependencies/readline/6.3/gnu//lib -Xlinker -export-dynamic
>>
>>
>> Below are the warnings and error I get:
>> /app/COST/python/2.7.9/gnu/lib/libpython2.7.a(dynload_shlib.o): In function `_PyImport_GetDynLoadFunc':
>> /app/COST/source/Python-2.7.9/Python/dynload_shlib.c:130: warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
>> /app/COST/python/2.7.9/gnu/lib/libpython2.7.a(posixmodule.o): In function `posix_tmpnam':
>> /app/COST/source/Python-2.7.9/./Modules/posixmodule.c:7575: warning: the use of `tmpnam_r' is dangerous, better use `mkstemp'
>> /app/COST/python/2.7.9/gnu/lib/libpython2.7.a(posixmodule.o): In function `posix_tempnam':
>> /app/COST/source/Python-2.7.9/./Modules/posixmodule.c:7522: warning: the use of `tempnam' is dangerous, better use `mkstemp'
>> /app/COST/python/2.7.9/gnu/lib/libpython2.7.a(posixmodule.o): In function `posix_initgroups':
>> /app/COST/source/Python-2.7.9/./Modules/posixmodule.c:4161: warning: Using 'initgroups' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
>> /app/COST/python/2.7.9/gnu/lib/libpython2.7.a(pwdmodule.o): In function `pwd_getpwall':
>> /app/COST/source/Python-2.7.9/./Modules/pwdmodule.c:165: warning: Using 'getpwent' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
>> /app/COST/python/2.7.9/gnu/lib/libpython2.7.a(pwdmodule.o): In function `pwd_getpwnam':
>> /app/COST/source/Python-2.7.9/./Modules/pwdmodule.c:139: warning: Using 'getpwnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
>> /app/COST/python/2.7.9/gnu/lib/libpython2.7.a(pwdmodule.o): In function `pwd_getpwuid':
>> /app/COST/source/Python-2.7.9/./Modules/pwdmodule.c:114: warning: Using 'getpwuid' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
>> /app/COST/python/2.7.9/gnu/lib/libpython2.7.a(pwdmodule.o): In function `pwd_getpwall':
>> /app/COST/source/Python-2.7.9/./Modules/pwdmodule.c:164: warning: Using 'setpwent' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
>> /app/COST/source/Python-2.7.9/./Modules/pwdmodule.c:176: warning: Using 'endpwent' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
>> /usr/bin/ld: dynamic STT_GNU_IFUNC symbol `strcmp' with pointer equality in `/usr/lib/../lib64/libc.a(strcmp.o)' can not be used when making an executable; recompile with -fPIE and relink with -pie
>> collect2: error: ld returned 1 exit status
>>
>>
>> Finally, here are the lines from my customize.py file:
>> compiler = 'cc'
>> define_macros += [('PARALLEL', '1')]
>> mpicompiler = 'cc'
>> mpilinker = mpicompiler
>>
>> libraries = ['xc']
>> scalapack = True
>>
>> mpi_library_dirs += ['/opt/cray/mpt/7.2.6/gni/mpich-gnu/51/lib']
>> library_dirs += ['/home/wwtomlin/xc/lib']
>> library_dirs += ['/opt/gcc/5.1.0/snos/lib64']
>> library_dirs += ['/opt/cray/libsci/13.2.0/GNU/5.1/x86_64/lib']
>> library_dirs += ['/app/COST/python/2.7.9/gnu/lib']
>>
>> mpi_include_dirs += ['/opt/cray/mpt/7.2.6/gni/mpich-gnu/51/include']
>> include_dirs += ['/home/wwtomlin/xc/include']
>> include_dirs += ['/opt/gcc/5.1.0/snos/include']
>> include_dirs += ['/opt/cray/libsci/13.2.0/GNU/5.1/x86_64/include']
>> include_dirs += ['/app/COST/python/2.7.9/gnu/include']
>>
>> define_macros += [('GPAW_NO_UNDERSCORE_CBLACS', '1')]
>> define_macros += [('GPAW_NO_UNDERSCORE_CSCALAPACK', '1’)]
>>
>>
>> My limited experience in this area tells me that there might be a problem with the way libc.a was compiled, but I’m note sure. I can’t do anything about that directly, but I could bring a problem to the attention of the system administrators and they’re usually pretty helpful getting things updated. Is that what I need to do?
>>
>> Thank you for any help,
>> Warren
>> PhD Student
>> Naval Postgraduate School
>>
>> _______________________________________________
>> gpaw-users mailing list
>> gpaw-users at listserv.fysik.dtu.dk
>> https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users
>
>
> _______________________________________________
> gpaw-users mailing list
> gpaw-users at listserv.fysik.dtu.dk
> https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users
More information about the gpaw-users
mailing list