PROPOSAL

Meta-prediction of machine learning performance


Supervisors: Veronika Cheplygina
Semester: Spring 2025
Tags: machine learning, medical imaging, data analysis, meta-research, reproducibility

Machine learning methods for medical imaging, for example segmentation of skin lesions or classification of lung cancer, are often evaluated on benchmark datasets such as ISIC, CheXpert, MIMIC-CXR and so forth. In such evaluation, researchers often compare the methods they propose, to state-of-the-art methods in the field, and report various performance metrics such as Dice score, AUC etc.

Due to the differences in dataset preprocessing, method parameters, and validation procedure (for example single train/test split, cross-validation), publications might report different results on what is essentially the same benchmark.

The goal of the project would be to create a dataset of reported experiments in ML papers at recent medical imaging conferences, analyze how the reported factors affect the reported performance, develop a method for predicting a method’s performance, and validate the developed methods on publications from a future conference.

Groups of 2+ students (from any study program, mixed groups are welcome) preferred.