The fundamental requirement for ground-breaking research is its reproducibility. A finding, no matter how provocative and exciting, has no value if it cannot be replicated. There is an increasing realization that the majority of experimental reports in the biological literature lack this fundamental element and the problem may be even greater for observational studies. This recognition has appropriately generated concern within the scientific and lay communities. And it should. But the fact that it is being debated widely indicates the strength of our system: an honest critique can drive change that will bring scientific advances which can actually be relied upon.
Highlighting this issue has drawn criticism from some within the scientific community, on the grounds that it should not have been discussed in public. Behind this criticism is a (legitimate) concern that a decline in public confidence may result in a decline research funding. However, given that research expenditure largely comes from the public purse, we scientists should be held accountable in the same way as other professions. Other scientists are concerned that cutting-edge science can be ‘difficult’ to reproduce, but cutting-edge science that is not robust enough to be reproduced by the investigators themselves when standard experimental practices are observed is of no value. Similarly other leading laboratories would expect to be able to reproduce a seminal finding. No ‘cutting-edge’ science is above the need for validation.
The ultimate responsibility for data quality rests with the investigator and their host institution. Other important stake-holders include the journals that publish scientific results, and the agencies that fund the research. It is heartening then to see the steps being taken to begin to address data integrity. Firstly at least two journals – Nature and Pigment Cell and Melanoma Research (PCMR) have re-written and published new Guidelines for Authors that focus on improving the quality and reliability of published data. Appropriately, the new demands from both journals will make it more onerous for authors. In instituting these changes, the Editors are attempting to decrease the sloppiness and lack of rigor that is prevalent in many papers published in biomedical journals. This is a significant step in terms of editorial responsibility, and will hopefully soon be followed by many other journals. Even with these changes, there is room for further improvement, for example, demonstrating a willingness to publish papers that contradict an earlier high-profile report, and to transparently identify publications that have transgressed Journal policies. However, the new Guidelines represent real progress, and the Editors of these journals are to be congratulated on the steps they have taken.
Funding agencies are also reviewing their policies in an attempt to improve data quality, for example, the National Institutes of Health are considering adding requirements for data validation to grant applications. Although some investigators regard such efforts with disdain, it is difficult to understand how scientists could object to their data and conclusions being confirmed! Yet, and often contrary to journal and funding agency policies, some investigators still refuse to make primary data sets available, so preventing independent analysis. Data verification provides the scientific community with a firm foundation upon which others can build. When these changes are implemented, this too will represent an important advance. Again, hopefully other funding agencies will soon follow.
As noted, however, the principal responsibility for data quality and integrity rests with the investigator and their host institution. The investigator alone knows whether experiments were performed and interpreted appropriately, or whether they represent a post-hoc construction woven together to ‘tell the best story’. The pressures on individual researchers to achieve fame and grant success are great, and this is an important driver in generation of poor quality, irreproducible research. Although some investigators are completely capable of balancing these conflicts and rigorously monitoring themselves, this is challenging, and many of us struggle to attain that balance, making institutional safeguards especially important. All the more so, since institutions themselves are typically recipients of public research funding, suggesting at least some level of public accountability. Institutions also bear responsibility for student and post-doctoral training and development, and for staff reviews and promotion. They can even place additional pressure on scientific staff to generate grant funding and thereby secure salary, laboratory space, students, and post-doctoral fellows. So it is appropriate to ask what research institutions are doing to address the issue of data irreproducibility emerging from within their walls. What new processes ensure the pressures that are imposed within the intensely-competitive research environment are being appropriately balanced? What new procedures ensure open reporting when investigators fail to comply with appropriate standards? What checks are in place to guarantee adequate supervision of junior staff regarding experimental design, data interpretation and presentation? Institutions have a responsibility to take a more aggressive role in transparently addressing these issues.
Institutions that routinely build a validation component into their key data are at a distinct advantage. This decreases unnecessary patent expenditure, as many filings would not be pursued; it gives the institutions an immediate advantage when negotiating with potential investors seeking to license their intellectual property; and it would negate the need for the investor to have findings confirmed, further raising the stature of the institution in the eyes of the commercial community.
The recent changes taking place in terms of editorial policy and NIH funding represent a genuine commitment to address the issue of data validation. Now it is time for academic research institutions to make a similar commitment.
Other Reference Sources
Prinz, F., Schlange, T. and Asadullah, K.A., (2011). Believe it or not: how much can we rely on published data on potential drug targets? Nature Reviews Drug Discovery 10, 712.
Begley, C.G., and Ellis, L. M., (2012). Drug development: Raise standards for preclinical cancer research. Nature 483, 531–533.
Ioannidis JPA (2005) Why Most Published Research Findings Are False. PLoS Med 2(8): e124. doi:10.1371/journal.pmed.0020124 http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fjournal.pmed.0020124