“Does peer review `work’ at all? A systematic review of all the available evidence on peer review concluded that `the practice of peer review is based on faith in its effects, rather than on facts‘.”
“Who is a peer?
“Somebody doing exactly the same kind of research (in which case he or she is probably a direct competitor)?
“Somebody in the same discipline?
“Somebody who is an expert on methodology?
“And what is review? Somebody saying `The paper looks all right to me‘, which is sadly what peer review sometimes seems to be. Or somebody poring all over the paper, asking for raw data, repeating analyses, checking all the references, and making detailed suggestions for improvement?
“Such a review is vanishingly rare…
“Robbie Fox, the great 20th century editor of the ‘Lancet’, who was no admirer of peer review, wondered whether anybody would notice if he were to swap the piles marked `publish’ and `reject’. He also joked that the Lancet had a system of throwing a pile of papers down the stairs and publishing those that reached the bottom. When I was editor of the ‘British Medical Journal’, I was challenged by two of the cleverest researchers in Britain to publish an issue of the journal comprised only of papers that had failed peer review and see if anybody noticed. I wrote back `How do you know I haven’t already done it?‘…
“Does peer review `work’ at all? A systematic review of all the available evidence on peer review concluded that `the practice of peer review is based on faith in its effects, rather than on facts‘.
(Jefferson T, Alderson P, Wager E, Davidoff F. Effects of editorial peer review: a systematic review. JAMA 2002;287: 2784-6)
–‘Peer review: a flawed process at the heart of science and journals’,
Journal of the Royal Society of Medicine, 2006 Apr; 99(4): 178–182
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1420798/
“…I have thirty-nine years of professional experience―twenty-six as a university professor, including fifteen at a major research university, and then thirteen as a researcher, writer, and editor―in close contact with scientists of various sorts, including some in the biological and physical sciences and many in the social sciences and demography.
“I have served as a peer reviewer for more than thirty professional journals and as a reviewer of research proposals for the National Science Foundation, the National Institutes of Health, and a number of large private foundations. I was the principal investigator of a major NSF-funded research project in the field of demography. So, I think I know something about how the system works.
“It does not work as outsiders seem to think.
“Peer review, on which lay people place great weight, varies from important — where the editors and the referees are competent and responsible — to a complete farce, where they are not. As a rule, not surprisingly, the process operates somewhere in the middle, being more than a joke but less than the nearly-flawless system of Olympian scrutiny that outsiders imagine it to be.
“Any journal editor who desires, for whatever reason, to knock down a submission can easily do so by choosing referees he knows full well will knock it down; likewise, he can easily obtain favorable referee reports. As I have always counseled young people whose work was rejected, seemingly on improper or insufficient grounds, the system is a crap shoot. Personal vendettas, ideological conflicts, professional jealousies, methodological disagreements, sheer self-promotion and a great deal of plain incompetence and irresponsibility are no strangers to the scientific world; indeed, that world is rife with these all-too-human attributes.
“In no event can peer review ensure that research is correct in its procedures or its conclusions. The history of every science is a chronicle of one mistake after another. In some sciences, these mistakes are largely weeded out in the course of time; in others, they persist for extended periods; and in some sciences, such as economics, actual scientific retrogression may continue for generations under the misguided belief that it is really progress.
“At any given time, consensus may exist about all sorts of matters in a particular science. In retrospect, however, that consensus is often seen to have been mistaken. As recently as the mid-1970s, for example, a scientific consensus existed among climatologists and scientists in related fields that the earth was about the enter a new ice age. Drastic proposals were made, such as exploding hydrogen bombs over the polar icecaps (to melt them) or damming the Bering Strait (to prevent cold Arctic water from entering the Pacific Ocean), to avert this impending disaster. Well-reputed scientists, not just uninformed wackos, made such proposals. How quickly we forget.
“Researchers who employ unorthodox methods or theoretical frameworks have great difficulty under modern conditions in getting their findings published in the “best” journals or, at times, in any scientific journal. Scientific innovators or creative eccentrics always strike the great mass of practitioners as nut cases―until it becomes impossible to deny their findings, a time that often comes only after one generation’s professional ring-masters have died off. Science is an odd undertaking: everybody strives to make the next breakthrough; yet when someone does, he is often greeted as if he were carrying the ebola virus.
“Too many people have too much invested in the reigning ideas; for those people an acknowledgment of their own idea’s bankruptcy is tantamount to an admission that they have wasted their lives. Often, perhaps to avoid cognitive dissonance, they never admit that their ideas were wrong. Most important, as a rule, in science as elsewhere, to get along, you must go along.
‘The In-Crowd’
“Research worlds, in their upper reaches, are pretty small. Leading researchers know all the major players and what everybody else is doing. They attend the same conferences, belong to the same societies, send their grad students to be postdocs in the other people’s labs, review one another’s work for the NSF, NIH, or other government funding organizations, and so forth. If you do not belong to this tight fraternity, it will prove very, very difficult for you to gain a hearing for your work, to publish in a “top” journal, to acquire a government grant, to receive an invitation to participate in a scientific-conference panel discussion, or to place your grad students in decent positions. The whole setup is tremendously incestuous; the interconnections are numerous, tight, and close.
“In this context, a bright young person needs to display cleverness in applying the prevailing orthodoxy, but it behooves him not to rock the boat by challenging anything fundamental or dear to the hearts of those who constitute the review committees for the NSF, NIH, and other funding organizations. Modern biological and physical science is, overwhelmingly, government-funded science. If your work, for whatever reason, does not appeal to the relevant funding agency’s bureaucrats and academic review committees, you can forget about getting any money to carry out your proposal.
“Recall the human frailties I mentioned previously; they apply just as much in the funding context as in the publication context. Indeed, these two contexts are themselves tightly linked: if you don’t get funding, you’ll never produce publishable work, and if you don’t land good publications, you won’t continue to receive funding.
“When your research implies a “need” for drastic government action to avert a looming disaster or to allay some dire existing problem, government bureaucrats and legislators (can you say “earmarks“?) are more likely to approve it. If the managers at the NSF, NIH, and other government funding agencies gave great amounts of money to scientists whose research implies that no disaster looms or no dire problem now exists or even that although a problem exists, no currently feasible government policy can do anything to solve it without creating even greater problems in the process, members of Congress would be much less inclined to throw money at the agency, with all the consequences that an appropriations cutback implies for bureaucratic thriving. No one has to explain all these things to the parties involved; they are not idiots, and they understand how the wheels are greased in their tight little worlds.
“Finally, we need to develop a much keener sense of what a scientist is qualified to talk about and what he is not qualified to talk about. Climatologists, for example, are qualified to talk about the science of climatology (though subject to all the intrusions upon pure science I have already mentioned). They are not qualified to say, however, that “we must act now” by imposing government “solutions” of some imagined sort. They are not professionally knowledgeable about what risk is better or worse for people to take; only the individuals who bear the risk can make that decision, because it’s a matter of personal preference, not a matter of science.
“Climatologists know nothing about cost/benefit cosiderations; indeed, most mainstream economists themselves are fundamentally misguided about such matters (adopting, for example, procedures and assumptions about the aggregation of individual valuations that lack a genuine scientific basis). Climate scientists are the best qualified people to talk about climate science, but they have no qualifications to talk about public policy, law, or individual values, rates of time preference, and degrees of risk aversion. In talking about desirable government action, they give the impression that they are either fools or charlatans, but they keep talking―worst of all, talking to doomsday-seeking journalists― nevertheless.
“In this connection, we might well bear in mind that the United Nations (and its committees and the bureaus it oversees) is no more a scientific organization than the U.S. Congress (and its committees and the bureaus it oversees). When decisions and pronouncements come forth from these political organizations, it makes sense to treat them as essentially political in origin and purpose. Politicians aren’t dumb, either―vicious, yes, but not dumb. One thing they know above everything else is how to stampede masses of people into approving or accepting ill-advised government actions that cost the people dearly in both their standard of living and their liberties in the long run.”
–‘Peer Review, Publication In Top Journals, Scientific Consensus, And So Forth’,
Robert Higgs, Independent Institute, May 7, 2007
https://www.independent.org/news/article.asp?id=1963
“One of the most important aspects of science is to actually communicate your results to your peers. This can be done through a variety of formal and informal ways. Emails are always popular, as per usual. And conferences are great places too to give quick status updates – although the most interesting conversations don’t happen during the presentations, but in the hallways and over dinners. But science has enjoyed centuries of tried-and-true tradition for widely disseminating research results: the journal article.
“Publishing a paper is considered, for good reason, a major accomplishment. It means that you’ve advanced the field, you’ve made your mark, you’ve upped your game. You’ve made enough progress and learned enough about nature that you think that your peers need to hear about it, and the publishers of the journal agree.
“But it doesn’t mean you’re right.
“The major hurdle for publishing a paper is peer review, when a (usually) anonymous member of your community (somebody who knows what they’re talking about, hopefully) reads the draft of your paper and offers critiques. Now these critiques are usually guided by the editor and the publishing standards of the journal. Most importantly, the reviewer has to determine if a significant scientific advancement was made, and whether that advancement is worthy enough for publication.
“What’s more, the reviewer has to determine if there are any major flaws in the paper. This is where things get interesting. It’s impossible for the reviewer to completely duplicate the experimental setup or observing program. And it’s usually very difficult to replicate the entire methods and analyses that were used to arrive at the conclusions. So the reviewer has to do their best job of following the logic and arguments laid out in the paper and seeing if they agree with the conclusions.
“But the reviewer isn’t getting paid. There’s no compensation involved except the kudos of the editor (who in gratitude will simply send you more papers to review in the future) and a sense of obligation that you’ve done your duty and service to the larger scientific enterprise.
“So in a system where the reviewer is volunteering their time, and can only do their best, what are we to make of peer review in general?
“Does it mean a paper is correct? No.
“Does it mean a paper is the final word? No.
“Does it mean that the scientific community has accepted the results? No.
“It means that one or a couple random reviewers thought the paper was interesting, new, and not obviously wrong.
“That’s it.
“So just because a paper has “passed” peer review, it doesn’t transmute into the scientific equivalent of Gospel Truth. Instead, peer review is the first step, not the last. Now that the paper is out in the wild, it’s time for it to be analyzed and picked apart – the real guts of scientific progress.”
–‘What Peer Review Actually Means’,
Paul M. Sutter, Forbes, Mar. 27, 2019
https://www.forbes.com/sites/paulmsutter/2019/03/27/what-peer-review-actually-means/?sh=6ba47b6c12d9
“One study showed that recently published articles, when resubmitted a few months later, are often rejected by the same journal – most of the reviewers did not detect that it was a resubmission, and the articles were frequently rejected due to “methodological flaws”, showing the volatility of reviewer decisions.”
“There are some age-old ways of cheating in the sciences — fabricating data or plagiarizing text — but apparently there’s a new cheat on the scene. Over the past two years more than 110 scientific papers have been retracted from people rigging the peer-review system, mostly by posing as independent reviewers and then evaluating their own articles.
“Cat Ferguson, Adam Marcus, and Ivan Oransky recently delved into the problem in a news feature published in the journal ‘Nature’. They describe several cases where researchers gave journals fake email addresses for other scientists — and then posed as these scientists to give positive reviews of their own papers. (The news feature also highlights how many journals’ reviewing-process software doesn’t ask for any identity verification.)
https://www.nature.com/articles/515480a
“These cases are part of what appears to be a recent spate of scientific malfeasance. So what’s going on here? Is this just a uniquely bad run? Or is this pointing to bigger flaws in the peer-review process itself?
“One of the most astounding cases of this kind of fake-identity technique recently led to the resignation of not just a scientist and a journal editor, but also the education minister of Taiwan. In July, scientific publisher SAGE announced that it was retracting a whopping 60 papers connected to Taiwanese researcher Peter Chen, in what appeared to be an elaborate work of fraud.
“This group of retractions is big enough for the history books. The 60 papers, published from 2010 to 2014 in the Journal of Vibration and Control, makes this one of the five biggest cases of retraction in science. (The dubious record is thought to be held by anesthesiologist Yoshitaka Fujii, who has 183 papers retracted or pending retraction.)
“SAGE’s ensuing, 14-month-long investigation showed that Chen had apparently created 130 fake email accounts of “assumed and fabricated identities” that created a “peer review and citation ring“. In other words, he seemed to be suggesting his own fake identities to the journal as reviewers of his papers (or sometimes posing as real people). And he may have used fake authors, too.
“This isn’t the first time that someone has been caught creating fake reviews. Experts can recall at least five other similar instances, including South Korean researcher Hyung-In Moon, who was caught in 2012 making up fake email addresses to review his own papers. He has had dozens of retractions so far.
“Chen isn’t even listed as an author on some of the papers that were retracted. Why would someone want to create a paper “written” by other people? Ivan Oransky, VP and global editorial director of MedPage Today and a co-author of the Nature feature, initially broke the story at the Retraction Watch blog and has been following cases of scientific misconduct for some time. In July, he said that these papers were likely Chen’s attempt to rack up citations of his own work.
“Chen has since resigned from his position at the National Pingtung University of Education. The editor of the journal where Chen’s work was published has left his job, too. And the scandal also led to the resignation of Taiwan’s education minister, who says he had been added as an author to several papers without his knowledge.
“Ideally, the peer-review process for scientific papers means that the journal will pick some experts to review someone’s paper. And they will keep those reviewers anonymous in order to solicit their honest opinions on the quality of the work. This is what happens at a place like Science or Nature.
“But at some journals, an editor might not know the specialized field very well and will ask the author of the paper for suggestions for reviewers. That’s what happened in Chen’s case — and it opened the door to serious malfeasance. However, if the journal had taken a look at those email addresses, it’s quite possible that they could have spotted what was going on. So allowing suggestions isn’t necessarily the problem.
“…On July 2, two high-profile papers on stem cells were retracted at the journal Nature. Meanwhile, a former Iowa State University HIV researcher just became one of the few people to ever face criminal charges for faking results from research conducted with federal dollars.
…
“…Some bad papers out there never get retracted.
“We have editors who stonewall, we have editors who are very stubborn about retracting“,
Oransky told me in July.
“We have scientists who threaten to sue if their paper is retracted. You have all these barriers to retraction.”
“One recent analysis documented several instances of papers with serious flaws (though no evidence of misconduct) that have never been retracted.
…”
–‘Scientists scammed at least 110 academic papers into publication using fake peer reviews’,
Susannah Locke, Vox, Dec. 7, 2014
See also:
‘Return to sender’,
Behind Lab Doors, 11 June 2015
https://behindlabdoors.com/responses/index.html
Share this:
Tags: Peer Review
Hector Heisler's Blog 
Leave a comment