Overview of the “Information Retrieval in Software Engineering” (IRSE) track at Forum for Information Retrieval 2024

The “Software Engineering Information Retrieval” (IRSE) track aims to devise solutions for the automated evaluation of code comments within a machine learning framework, with labels generated by both humans and large language models. Within this track, we offered a total of two tasks this year - i) a comment usefulness prediction task, and ii) a code quality estimation task.

The comment classification task involves discerning comments as either useful or not useful. The dataset includes 9,048 pairs of code comments and surrounding code snippets drawn from open-source C-based projects on GitHub and an additional dataset generated by teams employing large language models. In total, 12 teams representing various universities have contributed their experiments. These experiments were assessed through quantitative metrics, primarily the F1-Score, and qualitative evaluations based on the features developed, the supervised learning models employed, and their respective hyper-parameters. It is worth noting that labels generated by large language models introduce bias into the prediction model but lead to less over-fitted results.

The sub-track pertaining to code quality estimation was introduced this year. Given a problem description, and a list of large language model (LLM) generated software code, the objective of the task is to automatically estimate the functional correctness of each generated code. For the purpose of evaluation, each problem-solution pair is then ranked by these estimated probabilities of functional correctness, the quality of which is then reported with standard ranking performance measures.