A method for evaluating the interoperability of ontology classes in the behavioural and social sciences [version 1; peer review: awaiting peer review]

Background

Ontologies are frameworks for representing information that promote clarity, consistency and coherence, reduce the fragmentation of knowledge, and allow datasets and knowledge to be linked across studies, disciplines and domains. To enable this, it is important to identify how concepts of interest (‘classes’) are represented in different ontologies and evaluate the extent to which such classes align (i.e., are ‘interoperable’). This study aims to provide a method for doing this.

Methods

An automated tool using Meta’s Llama 3 language model was developed and used to compare artificial intelligence (AI) and human approaches to matching ontology classes. The automated tool was then integrated into a hybrid method for identifying classes that appear to refer to the same thing across pairs of ontologies. The method was evaluated by three behavioural scientists who used it to identify similar classes in two ontologies and provided feedback on their experience.

Results

The automated tool identified a larger number of potential matches than human-led review, so was used to generate a shortlist. The evaluation of the method produced mixed results. Users agreed which classes were identical or essentially the same across contexts, but none of the users identified similar classes that could be imported into an ontology without causing a contradiction or conflict. Users typically found using the method difficult, but many of the challenges related to using ontologies, rather than to the method specifically.

Conclusions

A combination of automated and human processes appears to be a feasible way to assess the interoperability of ontology classes. While further refinement is needed along with tools and resources that enable the use of ontologies by a broad range of researchers, the study provides a workable method for matching ontology classes in the behavioural and social sciences and offers a practical guide to support its implementation.