This resource contains 2 lessons
Image courtesy of the UC San Diego Library
(1) Scrutinizing science: Peer review. UC Museum of Paleontology of the University of California at Berkeley.
(2) In search of quality: The scientific peer review process. EUFIC – The European Food Information Council. A non-profit organisation for science-based information on food and health.
Scrutinizing Science: Peer Review
Peer review does the same thing for science that the “inspected by #7” sticker does for your t-shirt: provides assurance that someone who knows what they’re doing has double-checked it. In science, peer review typically works something like this:
A group of scientists completes a study and writes it up in the form of an article. They submit it to a journal for publication.
The journal’s editors send the article to several other scientists who work in the same field (i.e., the “peers” of peer review).
Those reviewers provide feedback on the article and tell the editor whether or not they think the study is of high enough quality to be published.
The authors may then revise their article and resubmit it for consideration.
Only articles that meet good scientific standards (e.g., acknowledge and build upon other work in the field, rely on logical reasoning and well-designed studies, back up claims with evidence, etc.) are accepted for publication.
Peer review and publication are time-consuming, frequently involving more than a year between submission and publication. The process is also highly competitive. For example, the highly-regarded journal Science accepts less than 8% of the articles it receives, and The New England Journal of Medicine publishes just 6% of its submissions.
Peer-reviewed articles provide a trusted form of scientific communication. Even if you are unfamiliar with the topic or the scientists who authored a particular study, you can trust peer-reviewed work to meet certain standards of scientific quality.
Since scientific knowledge is cumulative and builds on itself, this trust is particularly important. No scientist would want to base their own work on someone else’s unreliable study!
Peer-reviewed work isn’t necessarily correct or conclusive, but it does meet the standards of science. And that means that once a piece of scientific research passes through peer review and is published, science must deal with it somehow — perhaps by incorporating it into the established body of scientific knowledge, building on it further, figuring out why it is wrong, or trying to replicate its results.
– Scrutinizing science: Peer review. UC Museum of Paleontology of the University of California at Berkeley.
In search of quality: The scientific peer review process
Before a scientific assertion is made public it should be scrutinised for its credibility. Has the scientist drawn justifiable conclusions, based on the data available from sound scientific research?
The peer review process is a form of scientific quality control, where scientists open their research to the scrutiny of other experts in the field (peers).1
By reviewing and criticising each others’ work, scientists aim to ensure that only original and sound research is published and recognised.
How does it work?
When research is submitted for publication in a peer-reviewed journal, the journal invites several (usually two or more) independent experts to assess the credibility of the research.1
These experts consider the scientific methods, results and conclusions presented by the authors, asking themselves, if the science is technically sound, if the interpretation is consistent with the data, and if it is new, important or ground-breaking.2
Reviewers usually remain anonymous, are not paid for their assessment, and should not have any conflicts of interest in relation to the research. If a paper does not meet the requirements, based on the peer reviews, the editor can either reject it or deem it acceptable subject to adequate changes, allowing authors to react and revise their paper.
Why is it important?
The peer review process checks that a paper explains clearly how the research was carried out, so that it can be reproduced by others. It also verifies that the methodology is appropriate for the specific field and set of objectives.
Another crucial part of the review process is assessing the originality of new research and the accurate referencing of related published research, particularly if these contrast with the research at hand. The review is also useful for those whose work is being scrutinised; it allows them to fine-tune their manuscript before public release.2
A manuscript is seldom accepted for publication without at least a minor revision.
The review process essentially strives to separate fact from speculation and personal opinion.2
Peer-reviewed research is never beyond criticism however, and any conclusions drawn must be considered in the context of other studies. Ideally, experiments should be repeated to assess whether results can be reproduced; this is how findings are truly substantiated. The real validation, therefore, comes after publication.
Unfortunately, research results often find their way into the public domain without being peer reviewed, and are spread via newspapers, magazines, the internet, television and radio. They may be unpublished findings presented at press conferences, or published findings from a journal that does not use peer review.
Even journals that do use peer review contain some non-peer-reviewed content, such as editorials and letters to the editor. Both scientists and journalists should understand the meaning and importance of peer review and clarify whether or not research they discuss has been peer-reviewed. There are potentially enormous costs to both science and society from the promotion of scientifically weak or flawed research findings.
An imperfect process
The peer review process does not protect against misconduct. It can identify mistakes, but relies on honesty and, as a result, can fail to recognise deliberately fraudulent research. Various organisations have produced integrity guidelines on good research practice aiming to reduce such occurrences.3
On the other side, financial or personal concerns may bias a reviewer’s professional judgement and objectivity. It is vital to consider in advance any factors, which could lead to bias.3
According to the European Science Foundation, preventing and managing such conflicts of interest is crucial in ensuring equity and integrity.3
Sometimes concerns are raised about the influence of the funding body on the design of the study, or the interpretation or reporting of the research outcomes. The peer review process gives credence to research, because the paper has been independently checked and critically evaluated, including the correct scientific interpretation of the results on the basis of other existing evidence – no matter who funded the research.2
Inevitably, there are variations in standards between journals. A journal’s “Impact Factor” reflects how often its papers are cited in other peer-reviewed journals, and gives some indication of importance of the journal in its field – the higher the number, the greater the impact or influence.
The process and culture of checking each other’s work is ongoing in the scientific world. Once a paper has been published, further criticism can be made by the scientific community via letters to the journal editor, discussions at conferences, or direct exchange with the research team behind the study in question. Authors can justify their findings and flaws uncovered can be corrected or retracted.1,2
This is the nature of science; all work is open to critique by other scientists.
- Science Media Centre (2012). Peer review in a nutshell: http://www.sciencemediacentre.org/wp-content/uploads/2012/09/Peer-Review-in-a-Nutshell.pdf
- Sense About Science (2004). Peer Review and the acceptance of new scientific ideas. London: Sense About Science. http://www.senseaboutscience.org/data/files/resources/17/peerReview.pdf
- European Science Foundation (2011). European peer review guide integrating policies and practices into coherent procedures. Strasbourg: European Science Foundation. http://www.vr.se/download/18.2ab49299132224ae10680001647/European+Peer+Review+Guide.pdf