[ase-users] Database issues

[David Kleiven]

Dear ASE users,


we are working on a project that involves many DFT calculations and we use the database functionality to keep track of everything. Our calculations have a lot of external data for machine learning purposes. Currently, the only way we could find to "attach" such info to each structure is via the key_value_pair. To our knowledge, the key_value_pairs are duplicated in the database 1) They are stored as a serialized JSON string on each row and 2) Distributed in number_key_values and text_key_values. Hence, when you have for instance 5000 key_value_pairs it becomes cumbersome to maintain this duplication. Moreover, we have experienced cases where appending more key_value_pairs leads to errors and a corrupted database. We tested various solution and one way we found is to allow user defined tables (i.e. users can create tables with the same schema as number_key_values). Hence, big chunks of static data can be placed in those tables. One avoids duplication and appending dynamic key_value_pairs is no longer a problem. When you read back data from the database, all data from these external tables are automatically added to the AtomsRow object as if they were regular key_value_pairs.


The syntax for storing a separate table would be

db.write(atoms, tables={"some_table_name": dict_with_data}, ... regular key value pairs...)


and the same syntax for read. All external tables would be added to AtomsRow as if they where key_value_pairs.


Is this solution (separating out big chunks of data in separate tables) interesting to include in ASE via a class that inherits from SQLite3Database and provide this extra functionality in addition to everything that is supported by SQLite3Database? (Note: that in our case using the data field is not a good option as this is essentially a big binary chunk and it is therefore not so easy to 1) manually look at the data via external tools and 2) not so easy to update)

?

Cheers,

David Kleiven

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listserv.fysik.dtu.dk/pipermail/ase-users/attachments/20181123/451e2e1c/attachment.html>