Introduction

Overview

Teaching: 20 min
Exercises: 20 min
Questions
  • What is reproducible research?

  • What are the keys to reproducible research?

Objectives
  • Learn the four facets of reproducibility.

As you arrive, please make sure the following R packages are installed:

install.packages("tidyverse")

Motivating Reproducibility

Science retracts paper without agreement of lead author.

  • Journal retracted a study of how canvassers can sway people’s opinions about gay marriage. -Editor-in-Chief: Original survey data:
    • not made available for independent reproduction of results
    • Survey incentives misrepresented
    • Sponsorship statement false
  • Two grad students attempted to reproduce the study and could not. Concluded the data must have been fabricated. 538 story

  • Methods we’ll discuss today can’t prevent fraud, but they can make it easier to discover such issues.

Retracted, but not fraud

  • One researcher had seven papers retracted because of irreproducibility Retraction Watch
  • Couldn’t find the data Wiley
  • “Extensive” errors force retraction of lymphoma paper JRO
  • Many, many more Hackathon

Seizure study retracted after authors realized data were “Terribly mixed”

  • From the authors of Low Dose Lidocaine for Refractory Seizures in Preterm Neonates: “The article has been retracted at the request of the authors. After carefully re-examining the data presented in the article, they identified that data of two different hospitals got terribly mixed. The published results cannot be reproduced in accordance with scientific and clinical correctness.” IJP

Bad spreadsheet merge kills depression paper, quick fix resurrects it

  • The authors informed the journal that the merge of lab results and other survey data used in the paper resulted in an error regarding the identification codes.
  • Original conclusion : Lower levels of CSF IL-6 were associated with current depression and with future depression.
  • Revised conclusion: Higher levels of CSF IL-6 and IL-8 were associated with current depression.

Exercise: Motivating reproducibility

This is a two-part exercise:

Part 1: Analyze + Document

Complete the following tasks and write instructions / documentation for your collaborator to reproduce your work starting with the original dataset (data/gapminder-5060.csv):

  1. Visualize life expectancy over time for Canada in the 1950s and 1960s using a line plot.

  2. Something is clearly wrong with this plot! Turns out there’s a data error in the data file: life expectancy for Canada in the year 1957 is coded as 999999, it should actually be 69.96. Make this correction.

  3. Visualize life expectancy over time for Canada again, with the corrected data.

Stretch goal: Add lines for Mexico and United States.

Solution

  1. Import Data
gap_5060 <- read.csv("../data/gapminder-5060.csv")
  1. Load required pacakges
library("tidyverse")
## Loading tidyverse: ggplot2
## Loading tidyverse: tibble
## Loading tidyverse: tidyr
## Loading tidyverse: readr
## Loading tidyverse: purrr
## Loading tidyverse: dplyr
## Conflicts with tidy packages ----------------------------------------------
## filter(): dplyr, stats
## lag():    dplyr, stats
  1. Task 1 - Filter the data for Canada only:
gap_5060_CA <- gap_5060 %>% filter(country == "Canada")
  1. Task 1 - Visaualize:
ggplot(data = gap_5060_CA, aes(x = year, y = lifeExp)) +
   geom_line()

plot of chunk task_2_visualize Task 2. Something is clearly wrong with this plot! Turns out there’s a data error in the data file: life expectancy for Canada in the year 1957 is coded as 999999, it should actually be 69.96. Make this correction:

  • mutate for creating a new variables
  • replace for replacing a data entry in a specific location as determined by the logical statement (country == "Canada" & year == 1957)
gap_5060 <- gap_5060 %>%
  mutate(lifeExp = replace(lifeExp, (country == "Canada" & year == 1957), 69.96)) %>%
  as.data.frame()

Task 3. Visualize life expectancy over time for Canada again, with the corrected data. Exact same code as before, but note that the contents of gap_5060 are different as it has been updated in the previous task.

gap_5060_CA <- gap_5060 %>%
filter(country == "Canada")

ggplot(data = gap_5060_CA, aes(x = year, y = lifeExp)) +
  geom_line()

plot of chunk task_3 Task 3 - Stretch goal: Add lines for Mexico and United States.

  • %in% for logical operator testing if a country’s name is in the list provided
  • Same visualization code as before, only difference is the input dataset
gap_5060_NA <- gap_5060 %>%
  filter(country %in% c("Canada", "Mexico", "United States"))

ggplot(data = gap_5060_NA, aes(x = year, y = lifeExp, color = country)) +
  geom_line()

plot of chunk more_task_3

Part 2: Swap + discuss

Introduce yourself to your collaborator and tell them why you’re here.

  1. Swap instructions / documentation with your collaborator, and try to reproduce their work, first without talking to each other. If your collaborator does not have the software they need to reproduce your work, we encourage you to either help them install it or walk them through it on your computer in a way that would emulate the experience. (Remember, this could be part of the irreproducibility problem!)

  2. Then, talk to each other about challenges you faced (or didn’t face) or why you were or weren’t able to reproduce their work.

Reflection

Summary

Why we care

It is vital that scientific research be reproducible. This improves communication, openness, and transparency. We consider that there are four facets required to maximize reproducibility.

Four facets of reproducibility:

  1. Documentation: difference between binary files (e.g. docx) and text files and why text files are preferred for documentation.

  2. Organization: tools to organize your projects so that you don’t have a single folder with hundreds of files.

  3. Automation: the power of scripting to create automated data analyses.

  4. Dissemination: publishing is not the end of your analysis, rather it is a way station towards your future research and the future research of others.

Key Points

  • There are four facets to reproducibility: Documentation, Organization, Automation, Dissemination