Actionable advancements across scientific disciplines, including ML, are built on trust that requires sharing enough information about how a study is carried out to enable meaningful replication or reproduction by others. In principle, this process aims to facilitate self-correction across disciplines and weed-out literature that doesn't hold up to due scrutiny... but this doesn't always happen.

Watch this episode of Crash Course Statistics, hosted by Adriene Hill, and the accompanying video offered by Nature.

In this episode of Crash Course Statistics, Adriene Hill discusses the Replication Crisis across scientific disciplines. While the video refers to general study design replicability and reproducibility of statistical analysis, these concepts are largely applicable to machine learning contexts.

A few key points:

• Study Replication: Re-running studies to confirm results

• Reproducible Analyses: Analyses described such that others can repeat your methods on the same or similar a dataset.

• In 2016, a vast majority of researchers surveyed by Nature indicated that they believed the sciences were experiencing a study reproducibility crisis: an inability to reproduce findings from a large numbers of published studies across disciplines.

• Contributing factors to the replicability crisis are thought to include:

  • • • Publication bias, driven by reward-based structures in academic and industrial institutions.

      • Data Dredging (or "p-hacking") and lack of clarity around the role of statistical significance in context

        • • • "Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold ... statistical significance, does not measure the size of an effect or the importance of a result." - American Statistical Association

      • Lack of clear disclosures around data analysis and preprocessing methods.

        • • • While good papers contain detailed and explicit descriptions of their methods (i.e. disclosures), this is not always the standard across journals or across industry settings.

• Ideas around crisis mediation include:

  • • • More study replication, enabled by detailed methods disclosures and more funding for study replication

• More publication of quality research that yield null results to make p-hacking less enticing

• Promotion of data availability practices to emphasize reproducibility and transparency

Review this MIT Technology Review piece, written by Will Douglas Heaven.

In this piece, Will Douglas Heaven outlines the factors which contribute to the inability to replicate or reproduce findings in AI/ML literature, including lack of standardized data and code-sharing practices as well as competing interests of academic research and industry product showcasing.

A few key points

• Across the field of ML, concerns have been voiced that the ambition of novel experiments and model development supersede a dedication to sound and reproducible methodology.

• Lack of transparency and disclosures about model development methodology prevents new ML models and techniques from being adequately assessed for robustness, bias, and safety.

• Because the application of ML moves quickly from controlled environments to real-world scenarios that directly impact people’s lives, detailed disclosure practices that enable replication by different researchers across settings can expose errors and oversights sooner, making deployed ML models better for everyone.

• Known barriers to replication and strategies to address them:

• Lack of access to code

  • Code sharing practices, like those instituted by Papers with Code, are becoming more commonplace for ML literature, which enable greater opportunity for replication. However, sharing code alone is not enough to rerun an experiment. Without metadata and disclosures describing how models are trained and tuned, shared code can be useless. Ensuring that code is accompanied by detailed disclosures and metadata is essential for replication.

• Lack of access to data

  • • Data used in models of influence are often proprietary (i.e., Facebook user data) or subject to regulatory protections (i.e., protected health information). If researchers and ML practitioners are unable to share their data, it may instead be possible to provide directions so that others can build similar datasets, or to institute a process where a few independent auditors are given access to the data to verify results.

• Lack of access to hardware

  • • While the majority of ML research is run on computers that are available to the average lab, wealthy private tech companies are able to carry out further research on large and expensive collections of computers that few academic or public institutions have the resources to access. This is a problem in the context of replication; however, some point to the value of delayed replication of these models: ML methodologies that are inaccessible for other researchers to replicate in its early stages due to required computing power are often made more efficient (and thus more accessible for replication) as they are developed.

• Other Strategies:

• • Implement checklists of required materials for ML conference paper submissions, including code and detailed descriptions of experiments.

    • • When a checklist was introduced to the submission process for NeurIPS (a prominent conference on Neural Information Processing Systems) in 2019, the number of submissions including code jumped from less than 50% to around 75%.

  • Establishment of ML reproducibility challenges

  • Incentivizing replication efforts in academic and industry settings

  • Building replication work and disclosure documentation training into ML coursework

Cite as: Heaven WD. AI is wrestling with a replication crisis. MIT Technology Review. Published online November 12, 2020.

Disclosing dataset characteristics, and methods undertaken in dataset pre-processing and model development enables meaningful model replication and reproducibility efforts. In the absence of these disclosures, ML practitioners may contribute to an ongoing reproducibility and replication crisis in the ML literature and solution space (which is especially consequential in public health and healthcare contexts).