Reproducibility of Code-Understanding Large Language Models: A 217 Paper Analysis of ICSE '23 & SC '24

Reproducibility is essential to scientific progress: without it, results cannot be validated, built upon, or trusted. In machine learning and high-performance computing, complex software stacks mean that missing containerization, incomplete documentation, or unclear environment specifications can easily undermine workflow credibility and slow down collaboration. To evaluate the reproducibility of HPC/AI research papers in computer science and data science, a structured scorecard was used to assess the content of technical papers, the quality of the documentation and the reproducibility of the environment. This framework is informed by recent reproducibility benchmarks and HPC/AI research challenges