Difference between revisions of "Replication Crisis"

From FreeWiki
Jump to: navigation, search
Line 3: Line 3:
 
'''This article is closely connected to the article on the [[Decline Effect]], which is part of this crisis. For further explanation and material please see also there. '''
 
'''This article is closely connected to the article on the [[Decline Effect]], which is part of this crisis. For further explanation and material please see also there. '''
  
The '''replication crisis''' (or '''replicability crisis''' or '''reproducibility crisis''') is an ongoing (2019) methodological crisis in which it has been found that many scientific studies are difficult or impossible to replicate or reproduce. The replication crisis affects the social and life sciences most severely.<ref>{{Cite journal | doi = 10.1038/515009a| title = Metascience could rescue the 'replication crisis'| journal = Nature| volume = 515| issue = 7525| pages = 9| year = 2014| last1 = Schooler | first1 = J. W.| pmid=25373639| bibcode = 2014Natur.515....9S}}</ref><ref name="Why 'Statistical Significance' Is Often Insignificant">{{cite web|last1=Smith|first1=Noah|title=Why 'Statistical Significance' Is Often Insignificant|url=https://www.bloomberg.com/view/articles/2017-11-02/why-statistical-significance-is-often-insignificant|website=Bloomberg|accessdate=7 November 2017}}</ref> The crisis has long-standing roots; the phrase was coined in the early 2010s<ref>{{Cite journal | doi = 10.1177/1745691612465253| title = Editors' Introduction to the Special Section on Replicability in Psychological Science: A Crisis of Confidence?| journal = Perspectives on Psychological Science| volume = 7| issue = 6| pages = 528–530| year = 2012| last1=Pashler|first1=Harold |last2=Wagenmakers | first2=Eric Jan| pmid=26168108}}</ref> as part of a growing awareness of the problem. The replication crisis represents an important body of research in [[metascience]].<ref>{{cite web |last1=Fidler |first1=Fiona |last2=Wilcox |first2=John |title=Reproducibility of Scientific Results |url=https://plato.stanford.edu/entries/scientific-reproducibility/#MetaScieEstaMoniEvalReprCris |website=The Stanford Encyclopedia of Philosophy |publisher=Metaphysics Research Lab, Stanford University |accessdate=19 May 2019 |date=2018}}</ref>
+
The '''replication crisis''' (or '''replicability crisis''' or '''reproducibility crisis''') is an ongoing (2019) methodological crisis in which it has been found that many scientific studies are difficult or impossible to replicate or reproduce. The replication crisis affects the social and life sciences most severely.<ref>{{Cite journal | doi = 10.1038/515009a| title = Metascience could rescue the 'replication crisis'| journal = Nature| volume = 515| issue = 7525| pages = 9| year = 2014| last1 = Schooler | first1 = J. W.| pmid=25373639| bibcode = 2014Natur.515....9S}}</ref><ref name="Why 'Statistical Significance' Is Often Insignificant">{{cite web|last1=Smith|first1=Noah|title=Why 'Statistical Significance' Is Often Insignificant|url=https://www.bloomberg.com/view/articles/2017-11-02/why-statistical-significance-is-often-insignificant|website=Bloomberg|accessdate=7 November 2017}}</ref> The crisis has long-standing roots; the phrase was coined in the early 2010s<ref>{{Cite journal | doi = 10.1177/1745691612465253| title = Editors' Introduction to the Special Section on Replicability in Psychological Science: A Crisis of Confidence?| journal = Perspectives on Psychological Science| volume = 7| issue = 6| pages = 528–530| year = 2012| last1=Pashler|first1=Harold |last2=Wagenmakers | first2=Eric Jan| pmid=26168108}}</ref> as part of a growing awareness of the problem. The replication crisis represents an important body of research in metascience.<ref>{{cite web |last1=Fidler |first1=Fiona |last2=Wilcox |first2=John |title=Reproducibility of Scientific Results |url=https://plato.stanford.edu/entries/scientific-reproducibility/#MetaScieEstaMoniEvalReprCris |website=The Stanford Encyclopedia of Philosophy |publisher=Metaphysics Research Lab, Stanford University |accessdate=19 May 2019 |date=2018}}</ref>
  
 
Because the reproducibility of experiments is an essential part of the scientific method,<ref>Staddon, John (2017) Scientific Method: How science works, fails to work or pretends to work.  Taylor and Francis.</ref> the inability to replicate the studies of others has potentially grave consequences for many fields of science in which significant theories are grounded on unreproducible experimental work. The replication crisis has been particularly widely discussed in the field of psychology (and in particular, social psychology) and in [[medicine]], where a number of efforts have been made to re-investigate classic results, and to attempt to determine both the reliability of the results, and, if found to be unreliable, the reasons for the failure of replication.<ref>{{cite web|url=http://www.newyorker.com/tech/elements/the-crisis-in-social-psychology-that-isnt|title=The Crisis in Social Psychology That Isn't|author=Gary Marcus|publisher=The New Yorker|date=May 1, 2013}}</ref><ref>{{cite web|url=http://www.newyorker.com/magazine/2010/12/13/the-truth-wears-off|title=The Truth Wears Off|author=Jonah Lehrer|publisher=The New Yorker|date=December 13, 2010}}</ref>
 
Because the reproducibility of experiments is an essential part of the scientific method,<ref>Staddon, John (2017) Scientific Method: How science works, fails to work or pretends to work.  Taylor and Francis.</ref> the inability to replicate the studies of others has potentially grave consequences for many fields of science in which significant theories are grounded on unreproducible experimental work. The replication crisis has been particularly widely discussed in the field of psychology (and in particular, social psychology) and in [[medicine]], where a number of efforts have been made to re-investigate classic results, and to attempt to determine both the reliability of the results, and, if found to be unreliable, the reasons for the failure of replication.<ref>{{cite web|url=http://www.newyorker.com/tech/elements/the-crisis-in-social-psychology-that-isnt|title=The Crisis in Social Psychology That Isn't|author=Gary Marcus|publisher=The New Yorker|date=May 1, 2013}}</ref><ref>{{cite web|url=http://www.newyorker.com/magazine/2010/12/13/the-truth-wears-off|title=The Truth Wears Off|author=Jonah Lehrer|publisher=The New Yorker|date=December 13, 2010}}</ref>
Line 11: Line 11:
 
According to a 2016 poll of 1,500 scientists reported that 70% of them had failed to reproduce at least one other scientist's experiment (50% had failed to reproduce one of their own experiments).<ref>{{cite web|title=Is There a Reproducibility Crisis in Science?|url=https://www.scientificamerican.com/video/is-there-a-reproducibility-crisis-in-science/|publisher=Nature Video, Scientific American|date=28 May 2016|accessdate=15 August 2019}}</ref> In 2009, 2% of scientists admitted to falsifying studies at least once and 14% admitted to personally knowing someone who did. Misconducts were reported more frequently by medical researchers than others.<ref>{{cite journal|title=How Many Scientists Fabricate and Falsify Research? A Systematic Review and Meta-Analysis of Survey Data|first=Daniele|last=Fanelli|date=29 May 2009|journal=PLOS ONE|volume=4|issue=5|pages=e5738|doi=10.1371/journal.pone.0005738|pmid=19478950|pmc=2685008|bibcode=2009PLoSO...4.5738F}}</ref>
 
According to a 2016 poll of 1,500 scientists reported that 70% of them had failed to reproduce at least one other scientist's experiment (50% had failed to reproduce one of their own experiments).<ref>{{cite web|title=Is There a Reproducibility Crisis in Science?|url=https://www.scientificamerican.com/video/is-there-a-reproducibility-crisis-in-science/|publisher=Nature Video, Scientific American|date=28 May 2016|accessdate=15 August 2019}}</ref> In 2009, 2% of scientists admitted to falsifying studies at least once and 14% admitted to personally knowing someone who did. Misconducts were reported more frequently by medical researchers than others.<ref>{{cite journal|title=How Many Scientists Fabricate and Falsify Research? A Systematic Review and Meta-Analysis of Survey Data|first=Daniele|last=Fanelli|date=29 May 2009|journal=PLOS ONE|volume=4|issue=5|pages=e5738|doi=10.1371/journal.pone.0005738|pmid=19478950|pmc=2685008|bibcode=2009PLoSO...4.5738F}}</ref>
  
=== In psychology ===
+
=== Psychology ===
 
Several factors have combined to put psychology at the center of the controversy.<ref>{{cite web|last1=Achenbach|first1=Joel|title=No, science's reproducibility problem is not limited to psychology|url=https://www.washingtonpost.com/news/speaking-of-science/wp/2015/08/28/no-sciences-reproducibility-problem-is-not-limited-to-psychology/|website=The Washington Post|accessdate=10 September 2015}}</ref> Much of the focus has been on the area of [[social psychology]],<ref>{{Cite news|url=https://www.nytimes.com/2017/10/18/magazine/when-the-revolution-came-for-amy-cuddy.html|title=When the Revolution Came for Amy Cuddy|last=Dominus|first=Susan|date=2017-10-18|work=The New York Times|access-date=2017-10-19|language=en-US|issn=0362-4331}}</ref> although other areas of psychology such as [[clinical psychology]],<ref>{{cite journal|last1=Leichsenring|first1=Falk|last2=Abbass|first2=Allan|author-link2=Allan Abbass|last3=Hilsenroth|first3=Mark J.|last4=Leweke|first4=Frank|last5=Luyten|first5=Patrick|last6=Keefe|first6=Jack R.|last7=Midgley|first7=Nick|last8=Rabung|first8=Sven|last9=Salzer|first9=Simone|last10=Steiner|first10=Christiane|title=Biases in research: risk factors for non-replicability in psychotherapy and pharmacotherapy research|journal=[[Psychological Medicine]]|year=April 2017|volume=47|issue=6|pages=1000–1011|doi=10.1017/S003329171600324X|pmid=27955715|url=http://discovery.ucl.ac.uk/1532689/}}</ref><ref>{{cite journal|last=Hengartner|first=Michael P.|title=Raising Awareness for the Replication Crisis in Clinical Psychology by Focusing on Inconsistencies in Psychotherapy Research: How Much Can We Rely on Published Findings from Efficacy Trials?|journal=[[Frontiers in Psychology]]|volume=9|pages=256|publisher=[[Frontiers Media]]|doi=10.3389/fpsyg.2018.00256|pmid=29541051|pmc=5835722|date=February 28, 2018}}</ref> [[developmental psychology]],<ref>{{cite journal|last1=Frank|first1=Michael C.|author-link1=Michael C. Frank|last2=Bergelson|first2=Elika|last3=Bergmann|first3=Christina|last4=Cristia|first4=Alejandrina|last5=Floccia|first5=Caroline|last6=Gervain|first6=Judit|last7=Hamlin|first7=J. Kiley|last8=Hannon|first8=Erin E.|last9=Kline|first9=Melissa|last10=Levelt|first10=Claartje|last11=Lew-Williams|first11=Casey|last12=Nazzi|first12=Thierry|last13=Panneton|first13=Robin|last14=Rabagliati|first14=Hugh|last15=Soderstrom|first15=Melanie|last16=Sullivan|first16=Jessica|last17=Waxman|first17=Sandra|author-link17=Sandra Waxman|last18=Yurovsky|first18=Daniel|title=A Collaborative Approach to Infant Research: Promoting Reproducibility, Best Practices, and Theory‐Building|journal=[[Infancy (journal)|Infancy]]|volume=22|issue=4|pages=421–435|date=9 March 2017|url=http://callab.uchicago.edu/papers/fetal-infancy-inpress.pdf|doi=10.1111/infa.12182|access-date=19 December 2018|hdl=10026.1/9942}}</ref> and [[educational research]] have also been implicated.<ref>{{cite journal|last=Tyson|first=Charlie|title=Failure to Replicate|journal=[[Inside Higher Ed]]|date=14 August 2014|url=https://www.insidehighered.com/news/2014/08/14/almost-no-education-research-replicated-new-article-shows|access-date=19 December 2018}}</ref><ref>{{cite journal|last1=Makel|first1=Matthew C.|last2=Plucker|first2=Jonathan A.|author-link2=Jonathan A. Plucker|title=Facts Are More Important Than Novelty: Replication in the Education Sciences|journal=[[Educational Researcher]]|date=1 August 2014|volume=43|issue=6|pages=304–316|doi=10.3102/0013189X14545513|access-date=19 December 2018|url=https://journals.sagepub.com/stoken/rbtfl/w5mrNxPVD8zSg/full}}</ref> According to a 2018 survey of 200 meta-analyses, "psychological research is, on average, afflicted with low statistical power".<ref>{{Cite journal|last=Stanley|first=T. D.|last2=Carter|first2=Evan C.|last3=Doucouliagos|first3=Hristos|date=2018|title=What meta-analyses reveal about the replicability of psychological research.|journal=Psychological Bulletin|language=en|volume=144|issue=12|pages=1325–1346|doi=10.1037/bul0000169|pmid=30321017|issn=1939-1455}}</ref>
 
Several factors have combined to put psychology at the center of the controversy.<ref>{{cite web|last1=Achenbach|first1=Joel|title=No, science's reproducibility problem is not limited to psychology|url=https://www.washingtonpost.com/news/speaking-of-science/wp/2015/08/28/no-sciences-reproducibility-problem-is-not-limited-to-psychology/|website=The Washington Post|accessdate=10 September 2015}}</ref> Much of the focus has been on the area of [[social psychology]],<ref>{{Cite news|url=https://www.nytimes.com/2017/10/18/magazine/when-the-revolution-came-for-amy-cuddy.html|title=When the Revolution Came for Amy Cuddy|last=Dominus|first=Susan|date=2017-10-18|work=The New York Times|access-date=2017-10-19|language=en-US|issn=0362-4331}}</ref> although other areas of psychology such as [[clinical psychology]],<ref>{{cite journal|last1=Leichsenring|first1=Falk|last2=Abbass|first2=Allan|author-link2=Allan Abbass|last3=Hilsenroth|first3=Mark J.|last4=Leweke|first4=Frank|last5=Luyten|first5=Patrick|last6=Keefe|first6=Jack R.|last7=Midgley|first7=Nick|last8=Rabung|first8=Sven|last9=Salzer|first9=Simone|last10=Steiner|first10=Christiane|title=Biases in research: risk factors for non-replicability in psychotherapy and pharmacotherapy research|journal=[[Psychological Medicine]]|year=April 2017|volume=47|issue=6|pages=1000–1011|doi=10.1017/S003329171600324X|pmid=27955715|url=http://discovery.ucl.ac.uk/1532689/}}</ref><ref>{{cite journal|last=Hengartner|first=Michael P.|title=Raising Awareness for the Replication Crisis in Clinical Psychology by Focusing on Inconsistencies in Psychotherapy Research: How Much Can We Rely on Published Findings from Efficacy Trials?|journal=[[Frontiers in Psychology]]|volume=9|pages=256|publisher=[[Frontiers Media]]|doi=10.3389/fpsyg.2018.00256|pmid=29541051|pmc=5835722|date=February 28, 2018}}</ref> [[developmental psychology]],<ref>{{cite journal|last1=Frank|first1=Michael C.|author-link1=Michael C. Frank|last2=Bergelson|first2=Elika|last3=Bergmann|first3=Christina|last4=Cristia|first4=Alejandrina|last5=Floccia|first5=Caroline|last6=Gervain|first6=Judit|last7=Hamlin|first7=J. Kiley|last8=Hannon|first8=Erin E.|last9=Kline|first9=Melissa|last10=Levelt|first10=Claartje|last11=Lew-Williams|first11=Casey|last12=Nazzi|first12=Thierry|last13=Panneton|first13=Robin|last14=Rabagliati|first14=Hugh|last15=Soderstrom|first15=Melanie|last16=Sullivan|first16=Jessica|last17=Waxman|first17=Sandra|author-link17=Sandra Waxman|last18=Yurovsky|first18=Daniel|title=A Collaborative Approach to Infant Research: Promoting Reproducibility, Best Practices, and Theory‐Building|journal=[[Infancy (journal)|Infancy]]|volume=22|issue=4|pages=421–435|date=9 March 2017|url=http://callab.uchicago.edu/papers/fetal-infancy-inpress.pdf|doi=10.1111/infa.12182|access-date=19 December 2018|hdl=10026.1/9942}}</ref> and [[educational research]] have also been implicated.<ref>{{cite journal|last=Tyson|first=Charlie|title=Failure to Replicate|journal=[[Inside Higher Ed]]|date=14 August 2014|url=https://www.insidehighered.com/news/2014/08/14/almost-no-education-research-replicated-new-article-shows|access-date=19 December 2018}}</ref><ref>{{cite journal|last1=Makel|first1=Matthew C.|last2=Plucker|first2=Jonathan A.|author-link2=Jonathan A. Plucker|title=Facts Are More Important Than Novelty: Replication in the Education Sciences|journal=[[Educational Researcher]]|date=1 August 2014|volume=43|issue=6|pages=304–316|doi=10.3102/0013189X14545513|access-date=19 December 2018|url=https://journals.sagepub.com/stoken/rbtfl/w5mrNxPVD8zSg/full}}</ref> According to a 2018 survey of 200 meta-analyses, "psychological research is, on average, afflicted with low statistical power".<ref>{{Cite journal|last=Stanley|first=T. D.|last2=Carter|first2=Evan C.|last3=Doucouliagos|first3=Hristos|date=2018|title=What meta-analyses reveal about the replicability of psychological research.|journal=Psychological Bulletin|language=en|volume=144|issue=12|pages=1325–1346|doi=10.1037/bul0000169|pmid=30321017|issn=1939-1455}}</ref>
  
Line 20: Line 20:
 
Thirdly, several effects in psychological science have been found to be difficult to replicate even before the current replication crisis. For example, the scientific journal ''[[Society for Judgment and Decision Making#Publications and journal|Judgment and Decision Making]]'' has published several studies over the years that fail to provide support for the [[unconscious thought theory]]. Replications appear particularly difficult when research trials are pre-registered and conducted by research groups not highly invested in the theory under questioning.
 
Thirdly, several effects in psychological science have been found to be difficult to replicate even before the current replication crisis. For example, the scientific journal ''[[Society for Judgment and Decision Making#Publications and journal|Judgment and Decision Making]]'' has published several studies over the years that fail to provide support for the [[unconscious thought theory]]. Replications appear particularly difficult when research trials are pre-registered and conducted by research groups not highly invested in the theory under questioning.
  
These three elements together have resulted in renewed attention for replication supported by psychologist [[Daniel Kahneman]].<ref>{{cite web|title=A New Etiquette for Replication|url=https://www.scribd.com/doc/225285909/Kahneman-Commentary|first=Daniel|last=Kahneman|via=Scribd}}</ref> Scrutiny of many effects have shown that several core beliefs are hard to replicate. A recent special edition of the journal ''Social Psychology'' focused on replication studies and a number of previously held beliefs were found to be difficult to replicate.<ref>{{cite journal|year=2014 |title= Link to issue<!-- NONE - reference to entire issue -->|url=http://www.psycontent.com/content/l67413865317/?p=3f8268e0307f4036906c158b1667c57a&pi=0 |journal=Social Psychology |publisher=Hogrefe Publishing |volume=45 |issue=3 |archive-url=https://archive.is/20140530230728/http://www.psycontent.com/content/l67413865317/?p=3f8268e0307f4036906c158b1667c57a&pi=0 |archive-date=30 May 2014}}</ref>  A 2012 special edition of the journal ''Perspectives on Psychological Science'' also focused on issues ranging from publication bias to null-aversion that contribute to the replication crises in psychology.<ref>[http://pps.sagepub.com/content/7/6.toc Table of Contents]</ref> In 2015, the first open empirical study of reproducibility in Psychology was published, called the [[Reproducibility Project]]. Researchers from around the world collaborated to replicate 100 empirical studies from three top Psychology journals. Fewer than half of the attempted replications were successful at producing statistically significant results in the expected directions, though most of the attempted replications did produce trends in the expected directions.<ref>{{Cite journal | last1 = Open Science Collaboration | title = Estimating the reproducibility of Psychological Science | doi = 10.1126/science.aac4716 | journal = Science | volume = 349 | issue = 6251 | pages = aac4716 | year = 2015 | pmid=26315443| url = http://eprints.keele.ac.uk/877/1/Open%20Science%20%28Science%20Pre-Print%29.pdf | hdl = 10722/230596 }}</ref>
+
These three elements together have resulted in renewed attention for replication supported by psychologist Daniel Kahneman.<ref>{{cite web|title=A New Etiquette for Replication|url=https://www.scribd.com/doc/225285909/Kahneman-Commentary|first=Daniel|last=Kahneman|via=Scribd}}</ref> Scrutiny of many effects have shown that several core beliefs are hard to replicate. A recent special edition of the journal ''Social Psychology'' focused on replication studies and a number of previously held beliefs were found to be difficult to replicate.<ref>{{cite journal|year=2014 |title= Link to issue<!-- NONE - reference to entire issue -->|url=http://www.psycontent.com/content/l67413865317/?p=3f8268e0307f4036906c158b1667c57a&pi=0 |journal=Social Psychology |publisher=Hogrefe Publishing |volume=45 |issue=3 |archive-url=https://archive.is/20140530230728/http://www.psycontent.com/content/l67413865317/?p=3f8268e0307f4036906c158b1667c57a&pi=0 |archive-date=30 May 2014}}</ref>  A 2012 special edition of the journal ''Perspectives on Psychological Science'' also focused on issues ranging from publication bias to null-aversion that contribute to the replication crises in psychology.<ref>[http://pps.sagepub.com/content/7/6.toc Table of Contents]</ref> In 2015, the first open empirical study of reproducibility in Psychology was published, called the [[Reproducibility Project]]. Researchers from around the world collaborated to replicate 100 empirical studies from three top Psychology journals. Fewer than half of the attempted replications were successful at producing statistically significant results in the expected directions, though most of the attempted replications did produce trends in the expected directions.<ref>{{Cite journal | last1 = Open Science Collaboration | title = Estimating the reproducibility of Psychological Science | doi = 10.1126/science.aac4716 | journal = Science | volume = 349 | issue = 6251 | pages = aac4716 | year = 2015 | pmid=26315443| url = http://eprints.keele.ac.uk/877/1/Open%20Science%20%28Science%20Pre-Print%29.pdf | hdl = 10722/230596 }}</ref>
  
 
Many research trials and meta-analyses are compromised by poor quality and conflicts of interest that involve both authors and professional advocacy organizations, resulting in many false positives regarding the effectiveness of certain types of psychotherapy.<ref>{{cite web | url=http://blogs.plos.org/mindthebrain/2014/04/15/meta-analyses-conducted-professional-organizations-trustworthy/ | title=Are meta analyses conducted by professional organizations more trustworthy? | work=Mind the Brain | date=April 15, 2014 | accessdate=September 13, 2016 | first=James | last=Coyne}}</ref>
 
Many research trials and meta-analyses are compromised by poor quality and conflicts of interest that involve both authors and professional advocacy organizations, resulting in many false positives regarding the effectiveness of certain types of psychotherapy.<ref>{{cite web | url=http://blogs.plos.org/mindthebrain/2014/04/15/meta-analyses-conducted-professional-organizations-trustworthy/ | title=Are meta analyses conducted by professional organizations more trustworthy? | work=Mind the Brain | date=April 15, 2014 | accessdate=September 13, 2016 | first=James | last=Coyne}}</ref>
Line 46: Line 46:
 
For these reasons the authors advocated that psychology is facing a disciplinary social dilemma, where the interests of the discipline are at odds with the interests of the individual researcher.
 
For these reasons the authors advocated that psychology is facing a disciplinary social dilemma, where the interests of the discipline are at odds with the interests of the individual researcher.
  
==== "Methodological terrorism" controversy ====
+
=== Medicine ===
With the replication crisis of psychology earning attention, Princeton University psychologist Susan Fiske drew controversy for calling out critics of psychology.<ref name=":02">{{Cite news|url=http://uk.businessinsider.com/susan-fiske-methodological-terrorism-2016-9?r=US&IR=T|title=Scientists are furious after a famous psychologist accused her peers of 'methodological terrorism'|work=Business Insider|access-date=2017-10-04|language=en}}</ref><ref>{{Cite news|url=https://www.psychologicalscience.org/publications/observer/obsonline/draft-of-observer-column-sparks-strong-social-media-response.html|title=Draft of Observer Column Sparks Strong Social Media Response|work=Association for Psychological Science|access-date=2017-10-04|language=en-US}}</ref><ref>{{Cite journal|last=Fiske|first=Susan T.|date=2016-10-31|title=A Call to Change Science's Culture of Shaming|url=https://www.psychologicalscience.org/observer/a-call-to-change-sciences-culture-of-shaming|journal=APS Observer|language=en-US|volume=29|issue=9}}</ref><ref>{{Cite news|url=http://nymag.com/scienceofus/2016/10/inside-psychologys-methodological-terrorism-debate.html|title=Inside Psychology's 'Methodological Terrorism' Debate|last=Singal|first=Jesse|work=Science of Us|access-date=2017-10-04|language=en}}</ref> She called these unnamed "adversaries" names such as "methodological terrorist" and "self-appointed data police", and said that criticism of psychology should only be expressed in private or through contacting the journals.<ref name=":02" /> Columbia University statistician and political scientist [[Andrew Gelman]], "well-respected among the researchers driving the replication debate", responded to Fiske, saying that she had found herself willing to tolerate the "dead paradigm" of faulty statistics and had refused to retract publications even when errors were pointed out.<ref name=":02" /><ref>{{Cite news|url=http://andrewgelman.com/2017/10/04/breaking-pnas-changes-slogan/|title=BREAKING . . . . . . . PNAS updates its slogan! - Statistical Modeling, Causal Inference, and Social Science|date=2017-10-04|work=Statistical Modeling, Causal Inference, and Social Science|access-date=2017-10-04|language=en-US}}</ref> He added that her tenure as editor has been abysmal and that a number of published papers edited by her were found to be based on extremely weak statistics; one of Fiske's own published papers had a major statistical error and "impossible" conclusions.<ref name=":02" />
 
 
 
=== In medicine ===
 
 
Out of 49 medical studies from 1990–2003, with more than 1000 citations, 45 claimed that studied therapy was effective. Out of these studies, 16% were contradicted by subsequent studies, 16% had found stronger effects than did subsequent studies, 44% were replicated, and 24% remained largely unchallenged.<ref>{{cite journal|title=Contradicted and initially stronger effects in highly cited clinical research|last=Ioannidis JA|date=13 July 2005|journal=JAMA|volume=294|issue=2|pages=218–228|doi=10.1001/jama.294.2.218|pmid=16014596}}</ref> The US [[Food and Drug Administration]] in 1977–1990 found flaws in 10–20% of medical studies.<ref>{{cite journal|title=Scientific data audit—A key management tool|first=J. Leslie|last=Glick|date=1 January 1992|volume=2|issue=3|pages=153–168|doi=10.1080/08989629208573811|journal=Accountability in Research}}</ref> In a paper published in 2012, Glenn Begley, a biotech consultant working at [[Amgen]], and Lee Ellis, at the University of Texas, argued that only 11% of the pre-clinical cancer studies could be replicated.<ref name="Begley-Lee2012">{{Cite journal|last1=Begley |first1=C. G. |last2=Ellis |first2=L. M. |year=2012 |title=Drug Development: Raise Standards for Preclinical Cancer Research |journal=Nature |volume=483 |issue=7391 |pages=531–533|doi=10.1038/483531a |pmid=22460880 |bibcode=2012Natur.483..531B }}</ref><ref name="Begley2013">{{Cite journal|last=Begley |first=C. G. |year=2013 |title=Reproducibility: Six red flags for suspect work |journal=Nature |volume=497 |issue=7450 |pages=433–434|doi=10.1038/497433a |bibcode=2013Natur.497..433B }}</ref>
 
Out of 49 medical studies from 1990–2003, with more than 1000 citations, 45 claimed that studied therapy was effective. Out of these studies, 16% were contradicted by subsequent studies, 16% had found stronger effects than did subsequent studies, 44% were replicated, and 24% remained largely unchallenged.<ref>{{cite journal|title=Contradicted and initially stronger effects in highly cited clinical research|last=Ioannidis JA|date=13 July 2005|journal=JAMA|volume=294|issue=2|pages=218–228|doi=10.1001/jama.294.2.218|pmid=16014596}}</ref> The US [[Food and Drug Administration]] in 1977–1990 found flaws in 10–20% of medical studies.<ref>{{cite journal|title=Scientific data audit—A key management tool|first=J. Leslie|last=Glick|date=1 January 1992|volume=2|issue=3|pages=153–168|doi=10.1080/08989629208573811|journal=Accountability in Research}}</ref> In a paper published in 2012, Glenn Begley, a biotech consultant working at [[Amgen]], and Lee Ellis, at the University of Texas, argued that only 11% of the pre-clinical cancer studies could be replicated.<ref name="Begley-Lee2012">{{Cite journal|last1=Begley |first1=C. G. |last2=Ellis |first2=L. M. |year=2012 |title=Drug Development: Raise Standards for Preclinical Cancer Research |journal=Nature |volume=483 |issue=7391 |pages=531–533|doi=10.1038/483531a |pmid=22460880 |bibcode=2012Natur.483..531B }}</ref><ref name="Begley2013">{{Cite journal|last=Begley |first=C. G. |year=2013 |title=Reproducibility: Six red flags for suspect work |journal=Nature |volume=497 |issue=7450 |pages=433–434|doi=10.1038/497433a |bibcode=2013Natur.497..433B }}</ref>
  
 
A 2016 article by [[John Ioannidis]], Professor of Medicine and of Health Research and Policy at Stanford University School of Medicine and a Professor of Statistics at Stanford University School of Humanities and Sciences, elaborated on "Why Most Clinical Research Is Not Useful".<ref name="Ioannidis2016">{{cite journal | last1 = Ioannidis | first1 = JPA | year = 2016 | title = Why Most Clinical Research Is Not Useful | journal = PLoS Med | volume = 13 | issue = 6| page = e1002049 | doi = 10.1371/journal.pmed.1002049 | pmid = 27328301 | pmc = 4915619 }}</ref> In the article Ioannidis laid out some of the problems and called for reform, characterizing certain points for medical research to be useful again; one example he made was the need for medicine to be "patient centered" (e.g. in the form of the [[Patient-Centered Outcomes Research Institute]]) instead of the current practice to mainly take care of "the needs of physicians, investigators, or sponsors". Ioannidis is known for his research focus on science itself since the 2005 paper "Why Most Published Research Findings Are False".<ref>{{cite journal|title=Why Most Published Research Findings Are False|journal=[[PLoS Medicine]]|date=August 1, 2005|issn=1549-1277|pmc=1182327|pmid=16060722|volume=2|issue=8|doi=10.1371/journal.pmed.0020124|first=John P. A.|last=Ioannidis|pages=e124}}</ref>
 
A 2016 article by [[John Ioannidis]], Professor of Medicine and of Health Research and Policy at Stanford University School of Medicine and a Professor of Statistics at Stanford University School of Humanities and Sciences, elaborated on "Why Most Clinical Research Is Not Useful".<ref name="Ioannidis2016">{{cite journal | last1 = Ioannidis | first1 = JPA | year = 2016 | title = Why Most Clinical Research Is Not Useful | journal = PLoS Med | volume = 13 | issue = 6| page = e1002049 | doi = 10.1371/journal.pmed.1002049 | pmid = 27328301 | pmc = 4915619 }}</ref> In the article Ioannidis laid out some of the problems and called for reform, characterizing certain points for medical research to be useful again; one example he made was the need for medicine to be "patient centered" (e.g. in the form of the [[Patient-Centered Outcomes Research Institute]]) instead of the current practice to mainly take care of "the needs of physicians, investigators, or sponsors". Ioannidis is known for his research focus on science itself since the 2005 paper "Why Most Published Research Findings Are False".<ref>{{cite journal|title=Why Most Published Research Findings Are False|journal=[[PLoS Medicine]]|date=August 1, 2005|issn=1549-1277|pmc=1182327|pmid=16060722|volume=2|issue=8|doi=10.1371/journal.pmed.0020124|first=John P. A.|last=Ioannidis|pages=e124}}</ref>
  
=== In marketing ===
+
=== Marketing ===
 
Marketing is another discipline with a "desperate need" for replication.<ref>{{Cite journal|title = The desperate need for replications|journal = Journal of Consumer Research|date = 2001-06-01|pages = 149–158|volume = 28|issue = 1|doi = 10.1086/321953|first = John E.|last = Hunter}}</ref> Many famous marketing studies fail to be repeated upon replication, a notable example being the "too-many-choices" effect, in which a high number of choices of product makes a consumer less likely to purchase.<ref>{{Cite journal|last=Armstrong|first=J. S.|last2=Green|first2=Kesten|date=30 January 2017|title=Guidelines for Science: Evidence and Checklists|url=http://repository.upenn.edu/cgi/viewcontent.cgi?article=1258&context=marketing_papers|journal=Working Paper|volume=|pages=|via=}}</ref> In addition to the previously mentioned arguments, replications studies in marketing are needed to examine the applicability of theories and models across countries and cultures, which is especially important because of possible influences of [[globalization]].<ref>{{Cite journal|title = Effects of Subcultural Differences on Country and Product Evaluations: A Replication Study|journal = Journal of Global Marketing|date = 2016-03-22|pages = 115–127|volume = 29|issue = 3|doi = 10.1080/08911762.2015.1138012|first1 = Thomas|last1 = Aichner|first2 = Paolo|last2 = Coletti|first3 = Cipriano|last3 = Forza|first4 = Urban|last4 = Perkmann|first5 = Alessio|last5 = Trentin}}</ref>
 
Marketing is another discipline with a "desperate need" for replication.<ref>{{Cite journal|title = The desperate need for replications|journal = Journal of Consumer Research|date = 2001-06-01|pages = 149–158|volume = 28|issue = 1|doi = 10.1086/321953|first = John E.|last = Hunter}}</ref> Many famous marketing studies fail to be repeated upon replication, a notable example being the "too-many-choices" effect, in which a high number of choices of product makes a consumer less likely to purchase.<ref>{{Cite journal|last=Armstrong|first=J. S.|last2=Green|first2=Kesten|date=30 January 2017|title=Guidelines for Science: Evidence and Checklists|url=http://repository.upenn.edu/cgi/viewcontent.cgi?article=1258&context=marketing_papers|journal=Working Paper|volume=|pages=|via=}}</ref> In addition to the previously mentioned arguments, replications studies in marketing are needed to examine the applicability of theories and models across countries and cultures, which is especially important because of possible influences of [[globalization]].<ref>{{Cite journal|title = Effects of Subcultural Differences on Country and Product Evaluations: A Replication Study|journal = Journal of Global Marketing|date = 2016-03-22|pages = 115–127|volume = 29|issue = 3|doi = 10.1080/08911762.2015.1138012|first1 = Thomas|last1 = Aichner|first2 = Paolo|last2 = Coletti|first3 = Cipriano|last3 = Forza|first4 = Urban|last4 = Perkmann|first5 = Alessio|last5 = Trentin}}</ref>
  
=== In economics ===
+
=== Economics ===
 
A 2016 study in the journal ''Science'' found that one-third of 18 experimental studies from two top-tier economics journals (''American Economic Review'' and the ''Quarterly Journal of Economics'') failed to be successfully replicated.<ref>{{Cite journal|last=Camerer|first=Colin F.|last2=Dreber|first2=Anna|last3=Forsell|first3=Eskil|last4=Ho|first4=Teck-Hua|last5=Huber|first5=Jürgen|last6=Johannesson|first6=Magnus|last7=Kirchler|first7=Michael|last8=Almenberg|first8=Johan|last9=Altmejd|first9=Adam|date=2016-03-25|title=Evaluating replicability of laboratory experiments in economics|journal=Science|language=en|volume=351|issue=6280|pages=1433–1436|doi=10.1126/science.aaf0918|issn=0036-8075|pmid=26940865|bibcode=2016Sci...351.1433C}}</ref><ref>{{Cite news|url=http://www.sciencemag.org/news/2016/03/about-40-economics-experiments-fail-replication-survey|title=About 40% of economics experiments fail replication survey|date=2016-03-03|work=Science |access-date=2017-10-25|language=en}}</ref> A 2017 study in the ''Economic Journal'' suggested that "the majority of the average effects in the empirical economics literature are exaggerated by a factor of at least 2 and at least one-third are exaggerated by a factor of 4 or more".<ref>{{Cite journal|last=Ioannidis|first=John P. A.|last2=Stanley|first2=T. D.|last3=Doucouliagos|first3=Hristos|date=2017-10-01|title=The Power of Bias in Economics Research|journal=The Economic Journal|language=en|volume=127|issue=605|pages=F236–F265|doi=10.1111/ecoj.12461|issn=1468-0297}}</ref>
 
A 2016 study in the journal ''Science'' found that one-third of 18 experimental studies from two top-tier economics journals (''American Economic Review'' and the ''Quarterly Journal of Economics'') failed to be successfully replicated.<ref>{{Cite journal|last=Camerer|first=Colin F.|last2=Dreber|first2=Anna|last3=Forsell|first3=Eskil|last4=Ho|first4=Teck-Hua|last5=Huber|first5=Jürgen|last6=Johannesson|first6=Magnus|last7=Kirchler|first7=Michael|last8=Almenberg|first8=Johan|last9=Altmejd|first9=Adam|date=2016-03-25|title=Evaluating replicability of laboratory experiments in economics|journal=Science|language=en|volume=351|issue=6280|pages=1433–1436|doi=10.1126/science.aaf0918|issn=0036-8075|pmid=26940865|bibcode=2016Sci...351.1433C}}</ref><ref>{{Cite news|url=http://www.sciencemag.org/news/2016/03/about-40-economics-experiments-fail-replication-survey|title=About 40% of economics experiments fail replication survey|date=2016-03-03|work=Science |access-date=2017-10-25|language=en}}</ref> A 2017 study in the ''Economic Journal'' suggested that "the majority of the average effects in the empirical economics literature are exaggerated by a factor of at least 2 and at least one-third are exaggerated by a factor of 4 or more".<ref>{{Cite journal|last=Ioannidis|first=John P. A.|last2=Stanley|first2=T. D.|last3=Doucouliagos|first3=Hristos|date=2017-10-01|title=The Power of Bias in Economics Research|journal=The Economic Journal|language=en|volume=127|issue=605|pages=F236–F265|doi=10.1111/ecoj.12461|issn=1468-0297}}</ref>
  
=== In sports science ===
+
=== Sports Science ===
 
A 2018 study took the field of exercise and sports science to task for insufficient replication studies, limited reporting of null results and trivial results, and insufficient research transparency.<ref>{{Cite journal|last=Halperin|first=Israel|last2=Vigotsky|first2=Andrew D.|last3=Foster|first3=Carl|last4=Pyne|first4=David B.|date=2018-02-01|title=Strengthening the Practice of Exercise and Sport-Science Research|journal=International Journal of Sports Physiology and Performance|volume=13|issue=2|pages=127–134|doi=10.1123/ijspp.2017-0322|issn=1555-0273|pmid=28787228}}</ref> Statisticians have criticized sports science for common use of a controversial statistical method called "[[magnitude-based inference]]" that has allowed sports scientists to extract apparently significant results from noisy data where ordinary hypothesis testing would have found none.<ref>{{Cite news|url=https://fivethirtyeight.com/features/how-shoddy-statistics-found-a-home-in-sports-research/|title=How Shoddy Statistics Found A Home In Sports Research|date=2018-05-16|work=FiveThirtyEight|access-date=2018-05-16|language=en-US}}</ref>
 
A 2018 study took the field of exercise and sports science to task for insufficient replication studies, limited reporting of null results and trivial results, and insufficient research transparency.<ref>{{Cite journal|last=Halperin|first=Israel|last2=Vigotsky|first2=Andrew D.|last3=Foster|first3=Carl|last4=Pyne|first4=David B.|date=2018-02-01|title=Strengthening the Practice of Exercise and Sport-Science Research|journal=International Journal of Sports Physiology and Performance|volume=13|issue=2|pages=127–134|doi=10.1123/ijspp.2017-0322|issn=1555-0273|pmid=28787228}}</ref> Statisticians have criticized sports science for common use of a controversial statistical method called "[[magnitude-based inference]]" that has allowed sports scientists to extract apparently significant results from noisy data where ordinary hypothesis testing would have found none.<ref>{{Cite news|url=https://fivethirtyeight.com/features/how-shoddy-statistics-found-a-home-in-sports-research/|title=How Shoddy Statistics Found A Home In Sports Research|date=2018-05-16|work=FiveThirtyEight|access-date=2018-05-16|language=en-US}}</ref>
  
=== In hydrology ===
+
=== Hydrology ===
A 2019 study in ''Scientific Data'' found that a small number of articles in [[hydrology]] and [[water resources]] journals could be reproduced due to data unavailability. The study "estimated, with 95% confidence, that results might be reproduced for only 0.6% to 6.8% of all 1,989 articles".<ref>{{Cite journal|last=Stagge|first=James H.|last2=Rosenberg|first2=David E.|last3=Abdallah|first3=Adel M.|last4=Akbar|first4=Hadia|last5=Attallah|first5=Nour A.|last6=James|first6=Ryan|date=2019-02-26|title=Assessing data availability and research reproducibility in hydrology and water resources|journal=Scientific Data|language=en|volume=6|pages=190030|doi=10.1038/sdata.2019.30|pmid=30806638|pmc=6390703|issn=2052-4463|bibcode=2019NatSD...690030S}}</ref>
+
A 2019 study in ''Scientific Data'' found that a small number of articles in hydrology and water resources journals could be reproduced due to data unavailability. The study "estimated, with 95% confidence, that results might be reproduced for only 0.6% to 6.8% of all 1,989 articles".<ref>{{Cite journal|last=Stagge|first=James H.|last2=Rosenberg|first2=David E.|last3=Abdallah|first3=Adel M.|last4=Akbar|first4=Hadia|last5=Attallah|first5=Nour A.|last6=James|first6=Ryan|date=2019-02-26|title=Assessing data availability and research reproducibility in hydrology and water resources|journal=Scientific Data|language=en|volume=6|pages=190030|doi=10.1038/sdata.2019.30|pmid=30806638|pmc=6390703|issn=2052-4463|bibcode=2019NatSD...690030S}}</ref></br>This study mainly related to the quality of documentation of origninal material which is obviously only given scarcely or not at all. With no raw data available the factual validity of the scientific articles cannot be estimated, but the scientific non-reproducability is not proven thereby.
 
 
=== Political repercussions ===
 
In the US, science's reproducibility crisis has become a topic of political contention, linked to the attempt to diminish regulations – e.g. of emissions of pollutants, with the argument that these regulations are based on non-reproducible science.<ref name="Oreskes">{{Cite journal |last=Oreskes |first=N. |year=2018 |title=Beware: Transparency rule is a trojan horse |journal=Nature |volume=557 |issue=7706 |page=469 |doi=10.1038/d41586-018-05207-9|pmid=29789751 |bibcode=2018Natur.557..469O }}</ref><ref name="FUTURES2">{{cite journal |last1=Andrea |first1=Saltelli |title=Why science's crisis should not become a political battling ground |journal=Futures |date=December 2018 |volume=104 |pages=85–90 |doi=10.1016/j.futures.2018.07.006}}</ref> Previous attempts with the same aim accused studies of being non-transparent.<ref name="Michaels">{{Cite book|first=D. |last=Michaels |title=Doubt is their product: How industry's assault on science threatens your health |publisher=Oxford University Press |year=2008}}</ref>
 
  
 
== Causes of the crisis ==
 
== Causes of the crisis ==
Line 78: Line 72:
 
* No single party is solely responsible, and no single solution will suffice.
 
* No single party is solely responsible, and no single solution will suffice.
 
   
 
   
 +
===High rate of publications==
 
In fact some predictions of a possible crisis in the quality control mechanism of science can be traced back several decades, especially among scholars in science and technology studies (STS). Derek de Solla Price – considered the father of scientometrics – predicted that science could reach 'senility' as a result of its own exponential growth.<ref name="DeSollaPrice1963">{{cite book|author1=De Solla Price|author2=Derek J.|year=1963|title=Little science big science|publisher=Columbia University Press}}</ref> Some present day literature seems to vindicate this 'overflow' prophesy, lamenting at decay in both attention and quality.<ref name="Siebert-al2015">{{cite journal|author1=Siebert, S.|author2=Machesky, L. M.|author3=Insall, R. H.|lastauthoramp=y|year=2015|title=Overflow in science and its implications for trust|journal=eLife|volume=4|page=e10825|doi=10.7554/eLife.10825|pmid=26365552|pmc=4563216}}</ref><ref name="Della-Briotta-al2015">{{cite journal|author1=Della Briotta Parolo, P.|author2=Kumar Pan|author3=R. Ghosh|author4=R. Huberman|author5=B.A. Kimmo Kaski|author6=Fortunato, S.|year=2015|title=Attention decay in science|journal=Journal of Informetrics|volume=9|issue=4|pages=734–745|doi=10.1016/j.joi.2015.07.006|arxiv=1503.01881|bibcode=2015arXiv150301881D}}</ref>
 
In fact some predictions of a possible crisis in the quality control mechanism of science can be traced back several decades, especially among scholars in science and technology studies (STS). Derek de Solla Price – considered the father of scientometrics – predicted that science could reach 'senility' as a result of its own exponential growth.<ref name="DeSollaPrice1963">{{cite book|author1=De Solla Price|author2=Derek J.|year=1963|title=Little science big science|publisher=Columbia University Press}}</ref> Some present day literature seems to vindicate this 'overflow' prophesy, lamenting at decay in both attention and quality.<ref name="Siebert-al2015">{{cite journal|author1=Siebert, S.|author2=Machesky, L. M.|author3=Insall, R. H.|lastauthoramp=y|year=2015|title=Overflow in science and its implications for trust|journal=eLife|volume=4|page=e10825|doi=10.7554/eLife.10825|pmid=26365552|pmc=4563216}}</ref><ref name="Della-Briotta-al2015">{{cite journal|author1=Della Briotta Parolo, P.|author2=Kumar Pan|author3=R. Ghosh|author4=R. Huberman|author5=B.A. Kimmo Kaski|author6=Fortunato, S.|year=2015|title=Attention decay in science|journal=Journal of Informetrics|volume=9|issue=4|pages=734–745|doi=10.1016/j.joi.2015.07.006|arxiv=1503.01881|bibcode=2015arXiv150301881D}}</ref>
  
Line 84: Line 79:
 
Historian Philip Mirowski offered more recently a similar diagnosis in his 2011 book ''Science Mart'' (2011).<ref name="Mirowski">{{cite book|author=Mirowski, P.|year=2011|title=Science-Mart: Privatizing American Science|publisher=Harvard University Press}}</ref> 'Mart' is here a reference to the retail giant 'Walmart' and an allusion to the commodification of science. In the analysis of Mirowski, when science becomes a commodity being traded in a market, its quality collapses. Mirowski argues his case by tracing the decay of science to the decision of major corporations to close their in-house laboratories in order to outsource their work to universities, and subsequently to move their research away from universities to even cheaper contract research organizations (CRO).
 
Historian Philip Mirowski offered more recently a similar diagnosis in his 2011 book ''Science Mart'' (2011).<ref name="Mirowski">{{cite book|author=Mirowski, P.|year=2011|title=Science-Mart: Privatizing American Science|publisher=Harvard University Press}}</ref> 'Mart' is here a reference to the retail giant 'Walmart' and an allusion to the commodification of science. In the analysis of Mirowski, when science becomes a commodity being traded in a market, its quality collapses. Mirowski argues his case by tracing the decay of science to the decision of major corporations to close their in-house laboratories in order to outsource their work to universities, and subsequently to move their research away from universities to even cheaper contract research organizations (CRO).
  
 +
===Insufficient control===
 
The crisis of science's quality control system is affecting the use of science for policy. This is the thesis of a recent work by a group of STS scholars, who identify in 'evidence based (or informed) policy' a point of present tension.<ref name="FUTURES1">{{Cite journal|last1=Saltelli |first1=A. |last2=Funtowicz |first2=S. |year=2017 |title=What is science's crisis really about? |journal=Futures |volume=91 |pages=5–11 |doi=10.1016/j.futures.2017.05.010}}</ref><ref name="Benessia-al2016">{{Cite book|last1=Benessia |first1=A. |last2=Funtowicz |first2=S. |last3=Giampietro |first3=M. |last4=Guimarães Pereira |first4=A. |last5=Ravetz |first5=J. |last6=Saltelli |first6=A. |last7=Strand |first7=R. |last8=van der Sluijs |first8=J. |year=2016 |title=The Rightful Place of Science: Science on the Verge |publisher=Consortium for Science, Policy and Outcomes at Arizona State University|title-link=The Rightful Place of Science: Science on the Verge }}</ref><ref name="Letter">{{cite magazine|first1=Andrea |last1=Saltelli|first2=Jerome R. |last2=Ravetz|first3=Silvio |last3=Funtowicz|lastauthoramp=y|url=https://www.newscientist.com/letter/mg23030791-600-7-a-new-community-for-science/|title=A new community for science|magazine=New Scientist|issue=3079|p=52|date=25 June 2016}}</ref><ref name="FUTURES2" /> Economist Noah Smith suggests that a factor in the crisis has been the overvaluing of research in academia and undervaluing of teaching ability, especially in fields with few major recent discoveries.<ref name="Academic signaling and the post-truth world">{{cite web|last1=Smith|first1=Noah|title=Academic signaling and the post-truth world|url=http://noahpinionblog.blogspot.co.uk/2016/12/academic-signaling-and-post-truth-world.html|website=Noahpinion|publisher=Stony Brook University|accessdate=5 November 2017|date=2016-12-14}}</ref>
 
The crisis of science's quality control system is affecting the use of science for policy. This is the thesis of a recent work by a group of STS scholars, who identify in 'evidence based (or informed) policy' a point of present tension.<ref name="FUTURES1">{{Cite journal|last1=Saltelli |first1=A. |last2=Funtowicz |first2=S. |year=2017 |title=What is science's crisis really about? |journal=Futures |volume=91 |pages=5–11 |doi=10.1016/j.futures.2017.05.010}}</ref><ref name="Benessia-al2016">{{Cite book|last1=Benessia |first1=A. |last2=Funtowicz |first2=S. |last3=Giampietro |first3=M. |last4=Guimarães Pereira |first4=A. |last5=Ravetz |first5=J. |last6=Saltelli |first6=A. |last7=Strand |first7=R. |last8=van der Sluijs |first8=J. |year=2016 |title=The Rightful Place of Science: Science on the Verge |publisher=Consortium for Science, Policy and Outcomes at Arizona State University|title-link=The Rightful Place of Science: Science on the Verge }}</ref><ref name="Letter">{{cite magazine|first1=Andrea |last1=Saltelli|first2=Jerome R. |last2=Ravetz|first3=Silvio |last3=Funtowicz|lastauthoramp=y|url=https://www.newscientist.com/letter/mg23030791-600-7-a-new-community-for-science/|title=A new community for science|magazine=New Scientist|issue=3079|p=52|date=25 June 2016}}</ref><ref name="FUTURES2" /> Economist Noah Smith suggests that a factor in the crisis has been the overvaluing of research in academia and undervaluing of teaching ability, especially in fields with few major recent discoveries.<ref name="Academic signaling and the post-truth world">{{cite web|last1=Smith|first1=Noah|title=Academic signaling and the post-truth world|url=http://noahpinionblog.blogspot.co.uk/2016/12/academic-signaling-and-post-truth-world.html|website=Noahpinion|publisher=Stony Brook University|accessdate=5 November 2017|date=2016-12-14}}</ref>
  
Another major cause is [[publication bias]], the fact that positive results are more likely to get published than negative (or null) results. This may lead to the canonization of false facts <ref>{{cite journal |last1 = Nissen |first1 = Silas Boye |last2 = Magidson| first2 = Tali| last3 = Gross| first3 = Kevin | last4 = Bergstrom| first4 = Carl |title=Research: Publication bias and the canonization of false facts |journal=eLife |date=December 20, 2016 |volume=5 |pages=e21451 |url= https://elifesciences.org/articles/21451 |accessdate=9 June 2019 |doi=10.7554/eLife.21451 |arxiv= 1609.00494 }}</ref>.
+
===Publication bias===
 +
Another major cause is publication bias, the fact that positive results are more likely to get published than negative (or null) results. This may lead to the canonization of false facts <ref>{{cite journal |last1 = Nissen |first1 = Silas Boye |last2 = Magidson| first2 = Tali| last3 = Gross| first3 = Kevin | last4 = Bergstrom| first4 = Carl |title=Research: Publication bias and the canonization of false facts |journal=eLife |date=December 20, 2016 |volume=5 |pages=e21451 |url= https://elifesciences.org/articles/21451 |accessdate=9 June 2019 |doi=10.7554/eLife.21451 |arxiv= 1609.00494 }}</ref>.
 +
 
 +
===Principal reasons of the scientific paradigm===
 +
The biologist, philosopher and meta-scientist [[Rupert Sheldrake]] has pointed out that there is ample reason to doubt the belief in the full replicability and reliability of the hard sciences as well.<ref>Rupert Sheldrake: [https://www.sheldrake.org/essays/how-the-universal-gravitational-constant-varies?highlight=WyJwaHlzaWNhbCIsImNvbnN0YW50cyIsInBoeXNpY2FsIGNvbnN0YW50cyJd How the Universal Gravitational Constant Varies];
 +
[https://www.sheldrake.org/essays/the-replicability-crisis-in-science The Replicability Crisis in Science]; 
 +
Science Set Free: 10 Paths to New Discovery, ISBN-13: 978-0770436704, Publ. Deepak Chopra; 1st ed. (4. September 2012); The Science Delusion, ISBN-13: 978-1444727944, Publisher: Coronet (6 Dec. 2012) </ref>
 +
 
 +
In any case, the [[Decline_Effect|decline effect]] indicates general problems of replicability because it cannot be fully explained by the causes proposed so far. It was discovered by J.B.Rhine in the 20s of the last century, long before the current crisis and under completely different circumstances. Rhine found the effect in his own research, even in areas whose statistical validity he himself had previously verified.
 +
 
 +
==Public Response==
 +
=== Political Repercussions ===
 +
In the US, science's reproducibility crisis has become a topic of political contention, linked to the attempt to diminish regulations – e.g. of emissions of pollutants, with the argument that these regulations are based on non-reproducible science.<ref name="Oreskes">{{Cite journal |last=Oreskes |first=N. |year=2018 |title=Beware: Transparency rule is a trojan horse |journal=Nature |volume=557 |issue=7706 |page=469 |doi=10.1038/d41586-018-05207-9|pmid=29789751 |bibcode=2018Natur.557..469O }}</ref><ref name="FUTURES2">{{cite journal |last1=Andrea |first1=Saltelli |title=Why science's crisis should not become a political battling ground |journal=Futures |date=December 2018 |volume=104 |pages=85–90 |doi=10.1016/j.futures.2018.07.006}}</ref> Previous attempts with the same aim accused studies of being non-transparent.<ref name="Michaels">{{Cite book|first=D. |last=Michaels |title=Doubt is their product: How industry's assault on science threatens your health |publisher=Oxford University Press |year=2008}}</ref>
 +
 
 +
===Public discussion===
 +
Apart from studies and articles in scientific journals there has been little public discussion of the Replicability Crisis as yet and few reports in the mass media. Most of the discussions are related to statistical, economical and sociological reasons only.<ref>See all of the cited literature.</ref></br>The question whether there is a general problem or replicability behind these findings has not been tackled so far. And there has been very little effort in the hard sciences to deal with the replication issue as there seems to be a consense that the problem is only one of the social and life sciences.  
  
 
== Addressing the replication crisis ==
 
== Addressing the replication crisis ==

Revision as of 14:20, 2 September 2019

The following paragraph or article is based on the Wikipedia article „Replication Crisis“ from Wikipedia, as read on 31.8.2019, under the licence of Creative Commons CC-BY-SA 3.0 Unported (short version). A list of the authors is available on the respective page of Wikipedia. Changes are possible and probable.

This article is closely connected to the article on the Decline Effect, which is part of this crisis. For further explanation and material please see also there.

The replication crisis (or replicability crisis or reproducibility crisis) is an ongoing (2019) methodological crisis in which it has been found that many scientific studies are difficult or impossible to replicate or reproduce. The replication crisis affects the social and life sciences most severely.[1][2] The crisis has long-standing roots; the phrase was coined in the early 2010s[3] as part of a growing awareness of the problem. The replication crisis represents an important body of research in metascience.[4]

Because the reproducibility of experiments is an essential part of the scientific method,[5] the inability to replicate the studies of others has potentially grave consequences for many fields of science in which significant theories are grounded on unreproducible experimental work. The replication crisis has been particularly widely discussed in the field of psychology (and in particular, social psychology) and in medicine, where a number of efforts have been made to re-investigate classic results, and to attempt to determine both the reliability of the results, and, if found to be unreliable, the reasons for the failure of replication.[6][7]

Scope of the crisis

Overall

According to a 2016 poll of 1,500 scientists reported that 70% of them had failed to reproduce at least one other scientist's experiment (50% had failed to reproduce one of their own experiments).[8] In 2009, 2% of scientists admitted to falsifying studies at least once and 14% admitted to personally knowing someone who did. Misconducts were reported more frequently by medical researchers than others.[9]

Psychology

Several factors have combined to put psychology at the center of the controversy.[10] Much of the focus has been on the area of social psychology,[11] although other areas of psychology such as clinical psychology,[12][13] developmental psychology,[14] and educational research have also been implicated.[15][16] According to a 2018 survey of 200 meta-analyses, "psychological research is, on average, afflicted with low statistical power".[17]

Firstly, questionable research practices (QRPs) have been identified as common in the field.[18] Such practices, while not intentionally fraudulent, involve capitalizing on the gray area of acceptable scientific practices or exploiting flexibility in data collection, analysis, and reporting, often in an effort to obtain a desired outcome. Examples of QRPs include selective reporting or partial publication of data (reporting only some of the study conditions or collected dependent measures in a publication), optional stopping (choosing when to stop data collection, often based on statistical significance of tests), p-value rounding (rounding p-values down to 0.05 to suggest statistical significance), file drawer effect (nonpublication of data), post-hoc storytelling (framing exploratory analyses as confirmatory analyses), and manipulation of outliers (either removing outliers or leaving outliers in a dataset to cause a statistical test to be significant).[18][19][20][21] A survey of over 2,000 psychologists indicated that a majority of respondents admitted to using at least one QRP.[18] False positive conclusions, often resulting from the pressure to publish or the author's own confirmation bias, are an inherent hazard in the field, requiring a certain degree of skepticism on the part of readers.[22]

Secondly, psychology and social psychology in particular, has found itself at the center of several scandals involving outright fraudulent research, most notably the admitted data fabrication by Diederik Stapel[23] as well as allegations against others. However, most scholars acknowledge that fraud is, perhaps, the lesser contribution to replication crises.

Thirdly, several effects in psychological science have been found to be difficult to replicate even before the current replication crisis. For example, the scientific journal Judgment and Decision Making has published several studies over the years that fail to provide support for the unconscious thought theory. Replications appear particularly difficult when research trials are pre-registered and conducted by research groups not highly invested in the theory under questioning.

These three elements together have resulted in renewed attention for replication supported by psychologist Daniel Kahneman.[24] Scrutiny of many effects have shown that several core beliefs are hard to replicate. A recent special edition of the journal Social Psychology focused on replication studies and a number of previously held beliefs were found to be difficult to replicate.[25] A 2012 special edition of the journal Perspectives on Psychological Science also focused on issues ranging from publication bias to null-aversion that contribute to the replication crises in psychology.[26] In 2015, the first open empirical study of reproducibility in Psychology was published, called the Reproducibility Project. Researchers from around the world collaborated to replicate 100 empirical studies from three top Psychology journals. Fewer than half of the attempted replications were successful at producing statistically significant results in the expected directions, though most of the attempted replications did produce trends in the expected directions.[27]

Many research trials and meta-analyses are compromised by poor quality and conflicts of interest that involve both authors and professional advocacy organizations, resulting in many false positives regarding the effectiveness of certain types of psychotherapy.[28]

Although the British newspaper The Independent wrote that the results of the reproducibility project show that much of the published research is just "psycho-babble",[29] the replication crisis does not necessarily mean that psychology is unscientific.[30][31][32] Rather this process is a healthy if sometimes acrimonious part of the scientific process in which old ideas or those that cannot withstand careful scrutiny are pruned,[33][34] although this pruning process is not always effective.[35][36] The consequence is that some areas of psychology once considered solid, such as social priming, have come under increased scrutiny due to failed replications.[37]

Nobel laureate and professor emeritus in psychology Daniel Kahneman argued that the original authors should be involved in the replication effort because the published methods are often too vague.[38] Others such as Dr. Andrew Wilson disagree and argue that the methods should be written down in detail.[38] An investigation of replication rates in psychology in 2012 indicated higher success rates of replication in replication studies when there was author overlap with the original authors of a study[39] (91.7% successful replication rates in studies with author overlap compared to 64.6% success replication rates without author overlap).

Psychology replication rates

A report by the Open Science Collaboration in August 2015 that was coordinated by Brian Nosek estimated the reproducibility of 100 studies in psychological science from three high-ranking psychology journals.[40] Overall, 36% of the replications yielded significant findings (p value below 0.05) compared to 97% of the original studies that had significant effects. The mean effect size in the replications was approximately half the magnitude of the effects reported in the original studies.

The same paper examined the reproducibility rates and effect sizes by journal (Journal of Personality and Social Psychology [JPSP], Journal of Experimental Psychology: Learning, Memory, and Cognition [JEP:LMC], Psychological Science [PSCI]) and discipline (social psychology, cognitive psychology). Study replication rates were 23% for JPSP, 38% for JEP:LMC, and 38% for PSCI. Studies in the field of cognitive psychology had a higher replication rate (50%) than studies in the field of social psychology (25%).

An analysis of the publication history in the top 100 psychology journals between 1900 and 2012 indicated that approximately 1.6% of all psychology publications were replication attempts.[39] Articles were considered a replication attempt if the term "replication" appeared in the text. A subset of those studies (500 studies) was randomly selected for further examination and yielded a lower replication rate of 1.07% (342 of the 500 studies [68.4%] were actually replications). In the subset of 500 studies, analysis indicated that 78.9% of published replication attempts were successful. The rate of successful replication was significantly higher when at least one author of the original study was part of the replication attempt (91.7% relative to 64.6%).

A study published in 2018 in Nature Human Behaviour sought to replicate 21 social and behavioral science papers from Nature and Science, finding that only 13 could be successfully replicated.[41][42] Similarly, in a study conducted under the auspices of the Center for Open Science, a team of 186 researchers from 60 different laboratories (representing 36 different nationalities from 6 different continents) conducted replications of 28 classic and contemporary findings in psychology.[43] The focus of the study was not only on whether or not the findings from the original papers replicated, but also on the extent to which findings varied as a function of variations in samples and contexts. Overall, 14 of the 28 findings failed to replicate despite massive sample sizes. However, if a finding replicated, it replicated in most samples, while if a finding was not replicated, it failed to replicate with little variation across samples and contexts. This evidence is inconsistent with a popular explanation that failures to replicate in psychology are likely due to changes in the sample between the original and replication study.[44]

A disciplinary social dilemma

Highlighting the social structure that discourages replication in psychology, Brian D. Earp and Jim A. C. Everett enumerated five points as to why replication attempts are uncommon:[45][46]

  1. "Independent, direct replications of others' findings can be time-consuming for the replicating researcher
  2. "[Replications] are likely to take energy and resources directly away from other projects that reflect one's own original thinking
  3. "[Replications] are generally harder to publish (in large part because they are viewed as being unoriginal)
  4. "Even if [replications] are published, they are likely to be seen as 'bricklaying' exercises, rather than as major contributions to the field
  5. "[Replications] bring less recognition and reward, and even basic career security, to their authors"[47]

For these reasons the authors advocated that psychology is facing a disciplinary social dilemma, where the interests of the discipline are at odds with the interests of the individual researcher.

Medicine

Out of 49 medical studies from 1990–2003, with more than 1000 citations, 45 claimed that studied therapy was effective. Out of these studies, 16% were contradicted by subsequent studies, 16% had found stronger effects than did subsequent studies, 44% were replicated, and 24% remained largely unchallenged.[48] The US Food and Drug Administration in 1977–1990 found flaws in 10–20% of medical studies.[49] In a paper published in 2012, Glenn Begley, a biotech consultant working at Amgen, and Lee Ellis, at the University of Texas, argued that only 11% of the pre-clinical cancer studies could be replicated.[50][51]

A 2016 article by John Ioannidis, Professor of Medicine and of Health Research and Policy at Stanford University School of Medicine and a Professor of Statistics at Stanford University School of Humanities and Sciences, elaborated on "Why Most Clinical Research Is Not Useful".[52] In the article Ioannidis laid out some of the problems and called for reform, characterizing certain points for medical research to be useful again; one example he made was the need for medicine to be "patient centered" (e.g. in the form of the Patient-Centered Outcomes Research Institute) instead of the current practice to mainly take care of "the needs of physicians, investigators, or sponsors". Ioannidis is known for his research focus on science itself since the 2005 paper "Why Most Published Research Findings Are False".[53]

Marketing

Marketing is another discipline with a "desperate need" for replication.[54] Many famous marketing studies fail to be repeated upon replication, a notable example being the "too-many-choices" effect, in which a high number of choices of product makes a consumer less likely to purchase.[55] In addition to the previously mentioned arguments, replications studies in marketing are needed to examine the applicability of theories and models across countries and cultures, which is especially important because of possible influences of globalization.[56]

Economics

A 2016 study in the journal Science found that one-third of 18 experimental studies from two top-tier economics journals (American Economic Review and the Quarterly Journal of Economics) failed to be successfully replicated.[57][58] A 2017 study in the Economic Journal suggested that "the majority of the average effects in the empirical economics literature are exaggerated by a factor of at least 2 and at least one-third are exaggerated by a factor of 4 or more".[59]

Sports Science

A 2018 study took the field of exercise and sports science to task for insufficient replication studies, limited reporting of null results and trivial results, and insufficient research transparency.[60] Statisticians have criticized sports science for common use of a controversial statistical method called "magnitude-based inference" that has allowed sports scientists to extract apparently significant results from noisy data where ordinary hypothesis testing would have found none.[61]

Hydrology

A 2019 study in Scientific Data found that a small number of articles in hydrology and water resources journals could be reproduced due to data unavailability. The study "estimated, with 95% confidence, that results might be reproduced for only 0.6% to 6.8% of all 1,989 articles".[62]
This study mainly related to the quality of documentation of origninal material which is obviously only given scarcely or not at all. With no raw data available the factual validity of the scientific articles cannot be estimated, but the scientific non-reproducability is not proven thereby.

Causes of the crisis

In a work published in 2015, Glenn Begley and John Ioannidis offer five bullets as to summarize the present predicaments:[63]

  • Generation of new data/publications at an unprecedented rate.
  • Compelling evidence that the majority of these discoveries will not stand the test of time.
  • Causes: failure to adhere to good scientific practice and the desperation to publish or perish.
  • This is a multifaceted, multistakeholder problem.
  • No single party is solely responsible, and no single solution will suffice.

=High rate of publications

In fact some predictions of a possible crisis in the quality control mechanism of science can be traced back several decades, especially among scholars in science and technology studies (STS). Derek de Solla Price – considered the father of scientometrics – predicted that science could reach 'senility' as a result of its own exponential growth.[64] Some present day literature seems to vindicate this 'overflow' prophesy, lamenting at decay in both attention and quality.[65][66]

Philosopher and historian of science Jerome R. Ravetz predicted in his 1971 book Scientific Knowledge and Its Social Problems that science – in moving from the little science made of restricted communities of scientists to big science or techno-science – would suffer major problems in its internal system of quality control. Ravetz anticipated that modern science's system of rewarding scientists for research might become dysfunctional, the present 'publish or perish' challenge, creating perverse incentives to publish any findings however dubious. For Ravetz quality in science is maintained when there is a community of scholars linked by norms and standards, and a willingness to stand by these.

Historian Philip Mirowski offered more recently a similar diagnosis in his 2011 book Science Mart (2011).[67] 'Mart' is here a reference to the retail giant 'Walmart' and an allusion to the commodification of science. In the analysis of Mirowski, when science becomes a commodity being traded in a market, its quality collapses. Mirowski argues his case by tracing the decay of science to the decision of major corporations to close their in-house laboratories in order to outsource their work to universities, and subsequently to move their research away from universities to even cheaper contract research organizations (CRO).

Insufficient control

The crisis of science's quality control system is affecting the use of science for policy. This is the thesis of a recent work by a group of STS scholars, who identify in 'evidence based (or informed) policy' a point of present tension.[68][69][70][71] Economist Noah Smith suggests that a factor in the crisis has been the overvaluing of research in academia and undervaluing of teaching ability, especially in fields with few major recent discoveries.[72]

Publication bias

Another major cause is publication bias, the fact that positive results are more likely to get published than negative (or null) results. This may lead to the canonization of false facts [73].

Principal reasons of the scientific paradigm

The biologist, philosopher and meta-scientist Rupert Sheldrake has pointed out that there is ample reason to doubt the belief in the full replicability and reliability of the hard sciences as well.[74]

In any case, the decline effect indicates general problems of replicability because it cannot be fully explained by the causes proposed so far. It was discovered by J.B.Rhine in the 20s of the last century, long before the current crisis and under completely different circumstances. Rhine found the effect in his own research, even in areas whose statistical validity he himself had previously verified.

Public Response

Political Repercussions

In the US, science's reproducibility crisis has become a topic of political contention, linked to the attempt to diminish regulations – e.g. of emissions of pollutants, with the argument that these regulations are based on non-reproducible science.[75][71] Previous attempts with the same aim accused studies of being non-transparent.[76]

Public discussion

Apart from studies and articles in scientific journals there has been little public discussion of the Replicability Crisis as yet and few reports in the mass media. Most of the discussions are related to statistical, economical and sociological reasons only.[77]
The question whether there is a general problem or replicability behind these findings has not been tackled so far. And there has been very little effort in the hard sciences to deal with the replication issue as there seems to be a consense that the problem is only one of the social and life sciences.

Addressing the replication crisis

Replication has been referred to as "the cornerstone of science".[78][79] Replication studies attempt to evaluate whether published results reflect true findings or false positives. The integrity of scientific findings and reproducibility of research are important as they form the knowledge foundation on which future studies are built.

Metascience

Metascience is the use of scientific methodology to study science itself. Metascience seeks to increase the quality of scientific research while reducing waste. It is also known as "research on research" and "the science of science", as it uses research methods to study how research is done and where improvements can be made. Metascience concerns itself with all fields of research and has been described as "a bird's eye view of science."[80] In the words of John Ioannidis, "Science is the best thing that has happened to human beings ... but we can do it better."[81]

Meta-research continues to be conducted to identify the roots of the crisis and to address them. Methods of addressing the crisis include pre-registration of scientific studies and clinical trials as well as the founding of organizations such as CONSORT and the EQUATOR Network that issue guidelines for methodology and reporting. There are continuing efforts to reform the system of academic incentives, to improve the peer review process, to reduce the misuse of statistics, to combat bias in scientific literature, and to increase the overall quality and efficiency of the scientific process.

Tackling publication bias with pre-registration of studies

A recent innovation in scientific publishing to address the replication crisis is through the use of registered reports.[82][83] The registered report format requires authors to submit a description of the study methods and analyses prior to data collection. Once the method and analysis plan is vetted through peer-review, publication of the findings is provisionally guaranteed, based on whether the authors follow the proposed protocol. One goal of registered reports is to circumvent the publication bias toward significant findings that can lead to implementation of questionable research practices and to encourage publication of studies with rigorous methods.

The journal Psychological Science has encouraged the preregistration of studies and the reporting of effect sizes and confidence intervals.[84] The editor in chief also noted that the editorial staff will be asking for replication of studies with surprising findings from examinations using small sample sizes before allowing the manuscripts to be published.

Moreover, only a very small proportion of academic journals in psychology and neurosciences explicitly stated that they welcome submissions of replication studies in their aim and scope or instructions to authors.[85][86] This phenomenon does not encourage the reporting or even attempt on replication studies.

Emphasizing replication attempts in teaching

Based on coursework in experimental methods at MIT and Stanford, it has been suggested that methods courses in psychology emphasize replication attempts rather than original studies.[87][88] Such an approach would help students learn scientific methodology and provide numerous independent replications of meaningful scientific findings that would test the replicability of scientific findings. Some have recommended that graduate students should be required to publish a high-quality replication attempt on a topic related to their doctoral research prior to graduation.[46]

Reducing the p-value required for claiming significance of new results

Many publications require a p-value of p < 0.05 to claim statistical significance. The paper "Redefine statistical significance",[89] signed by a large number of scientists and mathematicians, proposes that in "fields where the threshold for defining statistical significance for new discoveries is P < 0.05, we propose a change to P < 0.005. This simple step would immediately improve the reproducibility of scientific research in many fields."

Their rationale is that "a leading cause of non-reproducibility (is that the) statistical standards of evidence for claiming new discoveries in many fields of science are simply too low. Associating 'statistically significant' findings with P < 0.05 results in a high rate of false positives even in the absence of other experimental, procedural and reporting problems."

Addressing the misinterpretation of p-values

Although statisticians are unanimous that use of the p < 0.05 provides weaker evidence than is generally appreciated, there is a lack of unanimity about what should be done about it. Some have advocated that Bayesian methods should replace p-values. This has not happened on a wide scale, partly because it is complicated, and partly because many users distrust the specification of prior distributions in the absence of hard data. A simplified version of the Bayesian argument, based on testing a point null hypothesis was suggested by Colquhoun (2014, 2017).[90][91] The logical problems of inductive inference were discussed in "The problem with p-values" (2016).[92]

The hazards of reliance on p-values were emphasized by pointing out that even observation of p = 0.001 was not necessarily strong evidence against the null hypothesis.[91] Despite the fact that the likelihood ratio in favour of the alternative hypothesis over the null is close to 100, if the hypothesis was implausible, with a prior probability of a real effect being 0.1, even the observation of p = 0.001 would have a false positive risk of 8 percent. It would not even reach the 5 percent level.

It was recommended[91] that the terms "significant" and "non-significant" should not be used. p-values and confidence intervals should still be specified, but they should be accompanied by an indication of the false positive risk. It was suggested that the best way to do this is to calculate the prior probability that would be necessary to believe in order to achieve a false positive risk of, say, 5%. The calculations can be done with R scripts that are provided,[91] or, more simply, with a web calculator.[93] This so-called reverse Bayesian approach, which was suggested by Matthews (2001),[94] is one way to avoid the problem that the prior probability is rarely known.

Encouraging larger sample sizes

To improve the quality of replications, larger sample sizes than those used in the original study are often needed.[95] Larger sample sizes are needed because estimates of effect sizes in published work are often exaggerated due to publication bias and large sampling variability associated with small sample sizes in an original study.[96][97][97][98] Further, using significance thresholds usually leads to inflated effects, because particularly with small sample sizes, only the largest effects will become significant.[99]

Sharing raw data in online repositories

Online repositories where data, protocols, and findings can be stored and evaluated by the public seek to improve the integrity and reproducibility of research. Examples of such repositories include the Open Science Framework, Registry of Research Data Repositories, and Psychfiledrawer.org. Sites like Open Science Framework offer badges for using open science practices in an effort to incentivize scientists. However, there has been concern that those who are most likely to provide their data and code for analyses are the researchers that are likely the most sophisticated.[100] John Ioannidis at Stanford University suggested that "the paradox may arise that the most meticulous and sophisticated and method-savvy and careful researchers may become more susceptible to criticism and reputation attacks by reanalyzers who hunt for errors, no matter how negligible these errors are".[100]

Funding for replication studies

In July 2016 the Netherlands Organisation for Scientific Research made €3 million available for replication studies. The funding is for replication based on reanalysis of existing data and replication by collecting and analysing new data. Funding is available in the areas of social sciences, health research and healthcare innovation.[101]

In 2013 the Laura and John Arnold Foundation funded the launch of The Center for Open Science with a $5.25 million grant and by 2017 had provided an additional $10 million in funding.[102] It also funded the launch of the Meta-Research Innovation Center at Stanford at Stanford University run by John Ioannidis and Steven Goodman to study ways to improve scientific research.[102] It also provided funding for the AllTrials initiative led in part by Ben Goldacre.[102]

Emphasize triangulation, not just replication

Marcus R. Munafò and George Davey Smith argue, in a piece published by Nature, that research should emphasize triangulation, not just replication. They claim that,

replication alone will get us only so far (and) might actually make matters worse ... We believe that an essential protection against flawed ideas is triangulation. This is the strategic use of multiple approaches to address one question. Each approach has its own unrelated assumptions, strengths and weaknesses. Results that agree across different methodologies are less likely to be artefacts. ... Maybe one reason replication has captured so much interest is the often-repeated idea that falsification is at the heart of the scientific enterprise. This idea was popularized by Karl Popper's 1950s maxim that theories can never be proved, only falsified. Yet an overemphasis on repeating experiments could provide an unfounded sense of certainty about findings that rely on a single approach. ... philosophers of science have moved on since Popper. Better descriptions of how scientists actually work include what epistemologist Peter Lipton called in 1991 "inference to the best explanation".[103]

References

  1. Schooler, J. W. (2014). "Metascience could rescue the 'replication crisis'". Nature. 515 (7525): 9. Bibcode:2014Natur.515....9S. doi:10.1038/515009a. PMID 25373639.
  2. Why 'Statistical Significance' Is Often Insignificant. Retrieved on 7 November 2017.
  3. Pashler, Harold; Wagenmakers, Eric Jan (2012). "Editors' Introduction to the Special Section on Replicability in Psychological Science: A Crisis of Confidence?". Perspectives on Psychological Science. 7 (6): 528–530. doi:10.1177/1745691612465253. PMID 26168108.
  4. Reproducibility of Scientific Results. Metaphysics Research Lab, Stanford University (2018). Retrieved on 19 May 2019.
  5. Staddon, John (2017) Scientific Method: How science works, fails to work or pretends to work. Taylor and Francis.
  6. Gary Marcus (May 1, 2013). The Crisis in Social Psychology That Isn't. The New Yorker.
  7. Jonah Lehrer (December 13, 2010). The Truth Wears Off. The New Yorker.
  8. Is There a Reproducibility Crisis in Science?. Nature Video, Scientific American (28 May 2016). Retrieved on 15 August 2019.
  9. Fanelli, Daniele (29 May 2009). "How Many Scientists Fabricate and Falsify Research? A Systematic Review and Meta-Analysis of Survey Data". PLOS ONE. 4 (5): e5738. Bibcode:2009PLoSO...4.5738F. doi:10.1371/journal.pone.0005738. PMC 2685008. PMID 19478950.
  10. No, science's reproducibility problem is not limited to psychology. Retrieved on 10 September 2015.
  11. Dominus, Susan (2017-10-18). "When the Revolution Came for Amy Cuddy". The New York Times. ISSN 0362-4331. Retrieved 2017-10-19.
  12. Leichsenring, Falk; Abbass, Allan; Hilsenroth, Mark J.; Leweke, Frank; Luyten, Patrick; Keefe, Jack R.; Midgley, Nick; Rabung, Sven; Salzer, Simone; Steiner, Christiane (April 2017). "Biases in research: risk factors for non-replicability in psychotherapy and pharmacotherapy research". Psychological Medicine. 47 (6): 1000–1011. doi:10.1017/S003329171600324X. PMID 27955715.
  13. Hengartner, Michael P. (February 28, 2018). "Raising Awareness for the Replication Crisis in Clinical Psychology by Focusing on Inconsistencies in Psychotherapy Research: How Much Can We Rely on Published Findings from Efficacy Trials?". Frontiers in Psychology. Frontiers Media. 9: 256. doi:10.3389/fpsyg.2018.00256. PMC 5835722. PMID 29541051.
  14. Frank, Michael C.; Bergelson, Elika; Bergmann, Christina; Cristia, Alejandrina; Floccia, Caroline; Gervain, Judit; Hamlin, J. Kiley; Hannon, Erin E.; Kline, Melissa; Levelt, Claartje; Lew-Williams, Casey; Nazzi, Thierry; Panneton, Robin; Rabagliati, Hugh; Soderstrom, Melanie; Sullivan, Jessica; Waxman, Sandra; Yurovsky, Daniel (9 March 2017). "A Collaborative Approach to Infant Research: Promoting Reproducibility, Best Practices, and Theory‐Building" (PDF). Infancy. 22 (4): 421–435. doi:10.1111/infa.12182. hdl:10026.1/9942. Retrieved 19 December 2018.
  15. Tyson, Charlie (14 August 2014). "Failure to Replicate". Inside Higher Ed. Retrieved 19 December 2018.
  16. Makel, Matthew C.; Plucker, Jonathan A. (1 August 2014). "Facts Are More Important Than Novelty: Replication in the Education Sciences". Educational Researcher. 43 (6): 304–316. doi:10.3102/0013189X14545513. Retrieved 19 December 2018.
  17. Stanley, T. D.; Carter, Evan C.; Doucouliagos, Hristos (2018). "What meta-analyses reveal about the replicability of psychological research". Psychological Bulletin. 144 (12): 1325–1346. doi:10.1037/bul0000169. ISSN 1939-1455. PMID 30321017.
  18. 18.0 18.1 18.2 John, Leslie K.; Loewenstein, George; Prelec, Drazen (2012-05-01). "Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling" (PDF). Psychological Science. 23 (5): 524–532. doi:10.1177/0956797611430953. ISSN 0956-7976. PMID 22508865.
  19. "The Nine Circles of Scientific Hell". Perspectives on Psychological Science. 7 (6): 643–644. 2012-11-01. doi:10.1177/1745691612459519. ISSN 1745-6916. PMID 26168124.
  20. Research misconduct - The grey area of Questionable Research Practices. Retrieved on 2015-11-13.
  21. Fiedler, Klaus; Schwarz, Norbert (2015-10-19). "Questionable Research Practices Revisited". Social Psychological and Personality Science. 7: 45–52. doi:10.1177/1948550615612150. ISSN 1948-5506.
  22. Simmons, Joseph; Nelson, Leif; Simonsohn, Uri (November 2011). "False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant". Psychological Science. 22 (11): 1359–1366. doi:10.1177/0956797611417632. ISSN 0956-7976. PMID 22006061.
  23. Shea, Christopher (13 November 2011). "Fraud Scandal Fuels Debate Over Practices of Social Psychology". The Chronicle of Higher Education.
  24. Kahneman, Daniel. A New Etiquette for Replication.
  25. "Link to issue". Social Psychology. Hogrefe Publishing. 45 (3). 2014. Archived from the original on 30 May 2014.
  26. Table of Contents
  27. Open Science Collaboration (2015). "Estimating the reproducibility of Psychological Science" (PDF). Science. 349 (6251): aac4716. doi:10.1126/science.aac4716. hdl:10722/230596. PMID 26315443.
  28. Coyne, James (April 15, 2014). Are meta analyses conducted by professional organizations more trustworthy?. Mind the Brain. Retrieved on September 13, 2016.
  29. Connor, Steve (27 August 2015). "Study reveals that a lot of psychology research really is just 'psycho-babble'". The Independent. London.
  30. Why Psychologists' Food Fight Matters (31 July 2014).
  31. Psychology Is Starting To Deal With Its Replication Problem (27 August 2015).
  32. Science Isn't Broken (19 August 2015).
  33. Etchells, Pete (28 May 2014). "Psychology's replication drive: it's not about you". The Guardian.
  34. Wagenmakers, Eric-Jan; Wetzels, Ruud; Borsboom, Denny; Maas, Han L. J. van der; Kievit, Rogier A. (2012-11-01). "An Agenda for Purely Confirmatory Research". Perspectives on Psychological Science. 7 (6): 632–638. doi:10.1177/1745691612463078. ISSN 1745-6916. PMID 26168122.
  35. Ioannidis, John P. A. (2012-11-01). "Why Science Is Not Necessarily Self-Correcting". Perspectives on Psychological Science. 7 (6): 645–654. doi:10.1177/1745691612464056. ISSN 1745-6916. PMID 26168125.
  36. Pashler, Harold; Harris, Christine R. (2012-11-01). "Is the Replicability Crisis Overblown? Three Arguments Examined". Perspectives on Psychological Science. 7 (6): 531–536. doi:10.1177/1745691612463401. ISSN 1745-6916. PMID 26168109.
  37. Bartlett, Tom (30 January 2013). "Power of Suggestion". The Chronicle of Higher Education.
  38. 38.0 38.1 Chambers, Chris (10 June 2014). "Physics envy: Do 'hard' sciences hold the solution to the replication crisis in psychology?". The Guardian.
  39. 39.0 39.1 Makel, Matthew C.; Plucker, Jonathan A.; Hegarty, Boyd (2012-11-01). "Replications in Psychology Research How Often Do They Really Occur?". Perspectives on Psychological Science. 7 (6): 537–542. doi:10.1177/1745691612460688. ISSN 1745-6916. PMID 26168110.
  40. Collaboration, Open Science (2015-08-28). "Estimating the reproducibility of psychological science" (PDF). Science. 349 (6251): aac4716. doi:10.1126/science.aac4716. hdl:10722/230596. ISSN 0036-8075. PMID 26315443.
  41. "The Science Behind Social Science Gets Shaken Up—Again". WIRED. Retrieved 2018-08-28.
  42. Camerer, Colin F.; Dreber, Anna; et al. (27 August 2018). "Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015" (PDF). Nature Human Behaviour. 2 (9): 637–644. doi:10.1038/s41562-018-0399-z. PMID 31346273.
  43. Klein, R.A. (2018). "Many Labs 2: Investigating Variation in Replicability Across Samples and Settings". Advances in Methods and Practices in Psychological Science. 1 (4): 443–490. doi:10.1177/2515245918810225.
  44. Witkowski, Tomasz (2019). "Is the glass half empty or half full? Latest results in the replication crisis in Psychology". Skeptical Inquirer. 43 (2): 5–6.
  45. See also Earp and Trafimow, 2015
  46. 46.0 46.1 Everett, Jim Albert Charlton; Earp, Brian D. (2015-01-01). "A tragedy of the (academic) commons: interpreting the replication crisis in psychology as a social dilemma for early-career researchers". Frontiers in Psychology. 6: 1152. doi:10.3389/fpsyg.2015.01152. PMC 4527093. PMID 26300832.
  47. Resolving the replication crisis in social psychology? A new proposal. Society for Personality and Social Psychology. Retrieved on 2015-11-18.
  48. Ioannidis JA (13 July 2005). "Contradicted and initially stronger effects in highly cited clinical research". JAMA. 294 (2): 218–228. doi:10.1001/jama.294.2.218. PMID 16014596.
  49. Glick, J. Leslie (1 January 1992). "Scientific data audit—A key management tool". Accountability in Research. 2 (3): 153–168. doi:10.1080/08989629208573811.
  50. Begley, C. G.; Ellis, L. M. (2012). "Drug Development: Raise Standards for Preclinical Cancer Research". Nature. 483 (7391): 531–533. Bibcode:2012Natur.483..531B. doi:10.1038/483531a. PMID 22460880.
  51. Begley, C. G. (2013). "Reproducibility: Six red flags for suspect work". Nature. 497 (7450): 433–434. Bibcode:2013Natur.497..433B. doi:10.1038/497433a.
  52. Ioannidis, JPA (2016). "Why Most Clinical Research Is Not Useful". PLoS Med. 13 (6): e1002049. doi:10.1371/journal.pmed.1002049. PMC 4915619. PMID 27328301.
  53. Ioannidis, John P. A. (August 1, 2005). "Why Most Published Research Findings Are False". PLoS Medicine. 2 (8): e124. doi:10.1371/journal.pmed.0020124. ISSN 1549-1277. PMC 1182327. PMID 16060722.
  54. Hunter, John E. (2001-06-01). "The desperate need for replications". Journal of Consumer Research. 28 (1): 149–158. doi:10.1086/321953.
  55. Armstrong, J. S.; Green, Kesten (30 January 2017). "Guidelines for Science: Evidence and Checklists". Working Paper.
  56. Aichner, Thomas; Coletti, Paolo; Forza, Cipriano; Perkmann, Urban; Trentin, Alessio (2016-03-22). "Effects of Subcultural Differences on Country and Product Evaluations: A Replication Study". Journal of Global Marketing. 29 (3): 115–127. doi:10.1080/08911762.2015.1138012.
  57. Camerer, Colin F.; Dreber, Anna; Forsell, Eskil; Ho, Teck-Hua; Huber, Jürgen; Johannesson, Magnus; Kirchler, Michael; Almenberg, Johan; Altmejd, Adam (2016-03-25). "Evaluating replicability of laboratory experiments in economics". Science. 351 (6280): 1433–1436. Bibcode:2016Sci...351.1433C. doi:10.1126/science.aaf0918. ISSN 0036-8075. PMID 26940865.
  58. "About 40% of economics experiments fail replication survey". Science. 2016-03-03. Retrieved 2017-10-25.
  59. Ioannidis, John P. A.; Stanley, T. D.; Doucouliagos, Hristos (2017-10-01). "The Power of Bias in Economics Research". The Economic Journal. 127 (605): F236–F265. doi:10.1111/ecoj.12461. ISSN 1468-0297.
  60. Halperin, Israel; Vigotsky, Andrew D.; Foster, Carl; Pyne, David B. (2018-02-01). "Strengthening the Practice of Exercise and Sport-Science Research". International Journal of Sports Physiology and Performance. 13 (2): 127–134. doi:10.1123/ijspp.2017-0322. ISSN 1555-0273. PMID 28787228.
  61. "How Shoddy Statistics Found A Home In Sports Research". FiveThirtyEight. 2018-05-16. Retrieved 2018-05-16.
  62. Stagge, James H.; Rosenberg, David E.; Abdallah, Adel M.; Akbar, Hadia; Attallah, Nour A.; James, Ryan (2019-02-26). "Assessing data availability and research reproducibility in hydrology and water resources". Scientific Data. 6: 190030. Bibcode:2019NatSD...690030S. doi:10.1038/sdata.2019.30. ISSN 2052-4463. PMC 6390703. PMID 30806638.
  63. Begley, C. G. & Ioannidis, J. P. (2015). "Reproducibility in Science: Improving the Standard for Basic and Preclinical Research". Circulation Research. 116 (1): 116–126. doi:10.1161/CIRCRESAHA.114.303819. PMID 25552691.
  64. De Solla Price; Derek J. (1963). Little science big science. Columbia University Press.
  65. Siebert, S.; Machesky, L. M. & Insall, R. H. (2015). "Overflow in science and its implications for trust". eLife. 4: e10825. doi:10.7554/eLife.10825. PMC 4563216. PMID 26365552.
  66. Della Briotta Parolo, P.; Kumar Pan; R. Ghosh; R. Huberman; B.A. Kimmo Kaski; Fortunato, S. (2015). "Attention decay in science". Journal of Informetrics. 9 (4): 734–745. arXiv:1503.01881. Bibcode:2015arXiv150301881D. doi:10.1016/j.joi.2015.07.006.
  67. Mirowski, P. (2011). Science-Mart: Privatizing American Science. Harvard University Press.
  68. Saltelli, A.; Funtowicz, S. (2017). "What is science's crisis really about?". Futures. 91: 5–11. doi:10.1016/j.futures.2017.05.010.
  69. Benessia, A.; Funtowicz, S.; Giampietro, M.; Guimarães Pereira, A.; Ravetz, J.; Saltelli, A.; Strand, R.; van der Sluijs, J. (2016). The Rightful Place of Science: Science on the Verge. Consortium for Science, Policy and Outcomes at Arizona State University.
  70. Saltelli, Andrea; Ravetz, Jerome R. & Funtowicz, Silvio (25 June 2016). "A new community for science". New Scientist. No. 3079. p. 52.
  71. 71.0 71.1 Andrea, Saltelli (December 2018). "Why science's crisis should not become a political battling ground". Futures. 104: 85–90. doi:10.1016/j.futures.2018.07.006.
  72. Academic signaling and the post-truth world. Stony Brook University (2016-12-14). Retrieved on 5 November 2017.
  73. Nissen, Silas Boye; Magidson, Tali; Gross, Kevin; Bergstrom, Carl (December 20, 2016). "Research: Publication bias and the canonization of false facts". eLife. 5: e21451. arXiv:1609.00494. doi:10.7554/eLife.21451. Retrieved 9 June 2019.
  74. Rupert Sheldrake: How the Universal Gravitational Constant Varies; The Replicability Crisis in Science; Science Set Free: 10 Paths to New Discovery, ISBN-13: 978-0770436704, Publ. Deepak Chopra; 1st ed. (4. September 2012); The Science Delusion, ISBN-13: 978-1444727944, Publisher: Coronet (6 Dec. 2012)
  75. Oreskes, N. (2018). "Beware: Transparency rule is a trojan horse". Nature. 557 (7706): 469. Bibcode:2018Natur.557..469O. doi:10.1038/d41586-018-05207-9. PMID 29789751.
  76. Michaels, D. (2008). Doubt is their product: How industry's assault on science threatens your health. Oxford University Press.
  77. See all of the cited literature.
  78. Moonesinghe, Ramal; Khoury, Muin J.; Janssens, A. Cecile J. W. (2007-02-27). "Most Published Research Findings Are False—But a Little Replication Goes a Long Way". PLoS Med. 4 (2): e28. doi:10.1371/journal.pmed.0040028. PMC 1808082. PMID 17326704.
  79. Simons, Daniel J. (2014-01-01). "The Value of Direct Replication". Perspectives on Psychological Science. 9 (1): 76–80. doi:10.1177/1745691613514755. ISSN 1745-6916. PMID 26173243.
  80. Ioannidis, John P. A.; Fanelli, Daniele; Dunne, Debbie Drake; Goodman, Steven N. (2015-10-02). "Meta-research: Evaluation and Improvement of Research Methods and Practices". PLOS Biology. 13 (10): –1002264. doi:10.1371/journal.pbio.1002264. ISSN 1545-7885. PMC 4592065. PMID 26431313.
  81. On communicating science and uncertainty: A podcast with John Ioannidis (8 December 2015). Retrieved on 20 May 2019.
  82. Registered Replication Reports. Association for Psychological Science. Retrieved on 2015-11-13.
  83. Chambers, Chris (2014-05-20). "Psychology's 'registration revolution'". The Guardian. Retrieved 2015-11-13.
  84. Lindsay, D. Stephen (2015-11-09). "Replication in Psychological Science". Psychological Science. 26 (12): 1827–32. doi:10.1177/0956797615616374. ISSN 0956-7976. PMID 26553013.
  85. Yeung, Andy W. K. (2017). "Do Neuroscience Journals Accept Replications? A Survey of Literature". Frontiers in Human Neuroscience. 11: 468. doi:10.3389/fnhum.2017.00468. ISSN 1662-5161. PMC 5611708. PMID 28979201.
  86. Martin, G. N.; Clarke, Richard M. (2017). "Are Psychology Journals Anti-replication? A Snapshot of Editorial Practices". Frontiers in Psychology. 8: 523. doi:10.3389/fpsyg.2017.00523. ISSN 1664-1078. PMC 5387793. PMID 28443044.
  87. Frank, Michael C.; Saxe, Rebecca (2012-11-01). "Teaching Replication". Perspectives on Psychological Science. 7 (6): 600–604. doi:10.1177/1745691612460686. ISSN 1745-6916. PMID 26168118.
  88. Grahe, Jon E.; Reifman, Alan; Hermann, Anthony D.; Walker, Marie; Oleson, Kathryn C.; Nario-Redmond, Michelle; Wiebe, Richard P. (2012-11-01). "Harnessing the Undiscovered Resource of Student Research Projects". Perspectives on Psychological Science. 7 (6): 605–607. doi:10.1177/1745691612459057. ISSN 1745-6916. PMID 26168119.
  89. Benjamin, Daniel. "Redefine statistical significance". PsyArXiv.
  90. Colquhoun, David (2015). "An investigation of the false discovery rate and the misinterpretation of p-values". Royal Society Open Science. 1 (3): 140216. arXiv:1407.5296. Bibcode:2014RSOS....140216C. doi:10.1098/rsos.140216. PMC 4448847. PMID 26064558.
  91. 91.0 91.1 91.2 91.3 Colquhoun, David (2017). "The reproducibility of research and the misinterpretation of p-values". Royal Society Open Science. 4 (12): 171085. doi:10.1098/rsos.171085. PMC 5750014. PMID 29308247.
  92. The problem with p-values. Aeon Magazine. Retrieved on 11 December 2016.
  93. Calculator for false positive risk (FPR). UCL.
  94. Matthews, R. A. J. (2001). "Why should clinicians care about Bayesian methods?". Journal of Statistical Planning and Inference. 94: 43–58. doi:10.1016/S0378-3758(00)00232-9.
  95. Maxwell, Scott E.; Lau, Michael Y.; Howard, George S. (2015). "Is psychology suffering from a replication crisis? What does "failure to replicate" really mean?". American Psychologist. 70 (6): 487–498. doi:10.1037/a0039400. PMID 26348332.
  96. IntHout, Joanna; Ioannidis, John P. A.; Borm, George F.; Goeman, Jelle J. (2015). "Small studies are more heterogeneous than large ones: a meta-meta-analysis". Journal of Clinical Epidemiology. 68 (8): 860–869. doi:10.1016/j.jclinepi.2015.03.017. PMID 25959635.
  97. 97.0 97.1 Button, Katherine S.; Ioannidis, John P. A.; Mokrysz, Claire; Nosek, Brian A.; Flint, Jonathan; Robinson, Emma S. J.; Munafò, Marcus R. (2013-05-01). "Power failure: why small sample size undermines the reliability of neuroscience". Nature Reviews Neuroscience. 14 (5): 365–376. doi:10.1038/nrn3475. ISSN 1471-003X. PMID 23571845.
  98. Greenwald, Anthony G. (1975). "Consequences of prejudice against the null hypothesis". Psychological Bulletin. 82 (1): 1–20. doi:10.1037/h0076157.
  99. Amrhein, Valentin; Korner-Nievergelt, Fränzi; Roth, Tobias (2017). "The earth is flat (p > 0.05): significance thresholds and the crisis of unreplicable research". PeerJ. 5: e3544. doi:10.7717/peerj.3544. PMC 5502092. PMID 28698825.
  100. 100.0 100.1 Ioannidis, John P. A. (2016). "Anticipating consequences of sharing raw data and code and of awarding badges for sharing". Journal of Clinical Epidemiology. 70: 258–260. doi:10.1016/j.jclinepi.2015.04.015. PMID 26163123.
  101. NWO makes 3 million available for Replication Studies pilot. Retrieved on 2 August 2016.
  102. 102.0 102.1 102.2 Apple, Sam (January 22, 2017). "The Young Billionaire Behind the War on Bad Science". Wired.
  103. Munafò, Marcus R.; Smith, George Davey (January 23, 2018). "Robust research needs many lines of evidence". Nature. 553 (7689): 399–401. Bibcode:2018Natur.553..399M. doi:10.1038/d41586-018-01023-3. PMID 29368721.

Further reading