In this presentation at the NIH Rehabilitation Research 2020: Envisioning a Functioning Future, Dr. Sook-Lei Liew discusses the challenges surrounding reproducibility and replicability in research. She also offers some practical solutions to solve these issues for achieving higher standards of research in the future.
Reproducibility, which refers to the ability of another person to re-create the structure of an experiment, often runs into problems when it comes to methods. This includes issues with manual input, inconsistent record-keeping, and differences across multiple researchers.
Replicability refers to the ability to achieve the same results every time an experiment is conducted. Logistical limitations can make sample sizes for studies end up underpowered, but if the results are significantly positive, it’s likely to be published due to the positive publication bias. This bias favors the publication of studies with significant positive results versus those without them.
How can these challenges be overcome? For reproducibility, we can adopt tools commonly used in data science for greater management and better results. Files that are consistent and well-organized can be machine-read for running data analyses, which makes your research much easier to reproduce.
For replicability, we can use open science to circumvent some of the issues caused by small sample sizes and publication bias by pooling together data sets. This would include data from both published and unpublished studies to create larger, more diverse samples to work with.
To access the full presentation for more details, visit the NIH Rehabilitation Research archive for a full recording here.
Below are a list of links to resources provided in the presentation:
Reproducible paper example:
Combining citizen science and deep learning to amplify expertise in neuroimaging (Keshavan et al., 2019)
Data Science: Resources in rehabilitation research
Center for Large Data Research and Data Sharing in Rehabilitation
2019 ASNR Symposium: Reliability and Reproducibility in Neurorehabilitation Research (hands-on tutorials and slides on Github)
Coursera, Udemy (introduction courses)
Open Data: Rehabilitation-Related Data Archives (NCMRR-funded)
Center for Large Data Research and Data in Rehabilitation
Archive of Data on Disability to Enable Policy and Research
Open Data: Community (Study-Specific) Brain Imaging
International Neuroimaging Data-Sharing Initiative
Neuroimaging Tools and Resources Collaboratory
Open Data: I want to share data
CLDR Rehabilitation-specific data sharing grant application
FAIR principles & reproducible methods for open science
Dr. Sook-Lei Liew is the director of the Neuroplasticity and Neurorehabilitation Laboratory at the University of Southern California. She also works as co-director of the SMART-VR Center, a collaborative effort dedicated to advancing neurorehabilitation with the use of Virtual Reality. You can read more about her work here.