Peering Behind The Veil Of Peer Review
From ASHRAE Journal, February 28, 2017
By Amy Wilson, Associate Editor
Have you ever wondered how the peer review process at ASHRAE Journal works? Or considered becoming a peer reviewer yourself? Journal staff sought to pull back the curtain and strip some of the mystery out of this behind-the-scenes process—and invite you to consider joining the discussion.
Since 1959, ASHRAE Journal has provided its readers thoughtful, technically sound content—and much of its credibility comes from the peer review process. But what is peer review?
At its most basic, peer review is an evaluation of an article by professionals who work in the same field. Peer-reviewed articles provide a trusted form of technical communication that meets certain standards of technical quality.
At ASHRAE Journal, an author submits an abstract or manuscript, which the editor assesses for meeting criteria outlined in the submissions guidelines. After criteria is met, the article is sent to three subject matter experts. The peer reviewers assess the technical merits of the paper and provide the author with recommendations for improvement.
Peer Review at ASHRAE Journal
So, what’s it like to be a peer reviewer at ASHRAE Journal? To find out, the Journal interviewed three peer reviewers—Lew Harriman, P.E., Fellow ASHRAE, Steve Taylor, P.E., Fellow ASHRAE, and Ramez Afify, HBDP, Member ASHRAE—about what it’s like to be a peer reviewer.
Not surprisingly, they differed somewhat in their outlook, but not in their overall conclusion that peer review is a vital tool for technical publishing.
“I felt it was a responsibility to step up and say ‘yes’ to being a peer reviewer,” recalls Harriman, who has been reviewing at the Journal for approximately 30 years.
He is also the author of many ASHRAE Journal articles and two books about humidity control and designing in hot and humid climates published by ASHRAE.
Taylor also has been a peer reviewer at the Journal for 30 years and is one of the columnists for ASHRAE Journal’s Engineer’s Notebook column. “It takes you into the subject more than just reading alone,” Taylor notes. “It makes you think the whole thing through, and forces you to focus a little better. Being a peer reviewer of articles makes you a better writer of articles.”
Afify—ASHRAE Journal author and relative newcomer to ASHRAE’s peer review process—agrees. “It’s exceedingly valuable. It gives readers that sense of confidence, that it’s been reviewed by other authors . And as a writer, I’ve written four articles for the Journal, and the peer review process helped me very much on that side.”
How Does It Work?
Typically, a technical committee chair will recommend peer reviewers. Sometimes, a Journal editor will reach out and ask you to consider being a peer reviewer. Past authors often become reviewers.
If you should receive a call, Lew Harriman suggests saying “yes”— even if you have reservations. “My advice is, everyone receiving the Journal is qualified to be a peer reviewer. If you don’t feel qualified or don’t understand the topic, you are a perfect person to be a peer reviewer! You as a technical person are the audience. If you write a comment of ‘I don’t understand’ in reference to what has been written, that is actually helpful. In fact, it takes some self-assurance to say, ‘I don’t understand what you are talking about.’ Remember, you are the audience. If you don’t understand this, it’s probably going to happen to others. So, everyone who receives this magazine is qualified to review.”
When you accept, the article will be sent with a peer review form for providing feedback on items such as technical accuracy, statement support, and overall value to new and experienced engineers. Afify explains, “I fill these out, and sometimes I give written comments in the text itself. Sometimes it’s [on] a separate form. The staff is always very friendly, very professional.” A panel of three peer reviewers reviews each article. The entire process is double-blind, so the author receives comments but does not know who wrote them. Similarly, the reviewers do not know who wrote the piece. The reviewers never see each other’s comments.
What Happens Next?
Reviews usually come back in about two weeks. Each reviewer provides comments on the piece and assigns it an overall score, ranging from 0 to 10. If the scores are low, the piece will be sent back to the author with comments for revision. Then the rewrite goes back through the peer review process again. Scores and comments are collected. At this point, if the paper passes review, the editor will schedule the article for publication. The whole process takes an average of two months—fairly brisk for a peer-reviewed publication.
Sometimes Reviewers Don’t Agree
Sometimes, the score is split with two reviewers rating it high and one rating it low. In these cases, a fourth reviewer is sometimes used or the staff sends the revised draft for second review only to the low-scoring reviewer. Usually, the low score is increased, often substantially.
“The more scrutiny an article receives, the more likely it is to be accurate and something people want to read,” says Taylor. “You have confidence when you read the article in the Journal that it is right, that it’s not simply an infomercial like so many other publications—that what’s being presented has been reviewed by technical people and is true.”
“Boiled down to its essence, the Journal is different from any other technical magazine,” says Harriman. “At one point the Journal changed its focus to become about, ‘How do we help people in the industry—engineers, building owners, etc.—do a good job?’ That was the magazine’s mission statement about 30 years ago, and I think we’ve lived up to it.”
As You Step into the Role
As a peer reviewer, you are playing a vital role. According to Harriman, “When I read a story, I think about, ‘What really happened? What was it and why? What happened in this building that made things different, and how can this story help? We are looking for experiences in the real world, experiences for engineers and building owners. It’s an applied technology journal versus deep foundational research.”
Afify says he appreciates articles that bring something new to the industry. “I enjoy it when a person brings up a point that nobody knows about. Some articles are back-to-basics. I would give an article more consideration if someone brings something new, such as new calculations, etc. To me, that should have a higher grade.”
Tips from a Peer Reviewer
As a veteran peer reviewer, Harriman has learned a few things about the process. Here are some of his suggestions if you are considering stepping into this role.
Don’t be afraid to accept. No, really. If you are a reader, you are qualified. And if you simply cannot accept, refer the Journal editor to a trusted colleague—even if that person hasn’t peer reviewed before. “Don’t let the fact that they aren’t an ‘anointed’ peer reviewer discourage you from asking,” Harriman suggests.
When reviewing, look for “relevant accuracy.” What does this mean to Harriman? “Sometimes engineers get wrapped up in accuracy—what’s the fifth decimal place and so on. Sometimes people write like that out of a suspicion that people won’t find them accurate. Sometimes the fifth decimal place doesn’t matter to the overall story. Remember you’re trying to help people focus on things that are most important. People aren’t as interested in the first law of thermodynamics as they are in, ‘Why did the toilet overflow?’ Look for detail that does not obscure the structure of the information and the thrust of the communication. Think about communicating across the desk with a respected colleague who doesn’t know what you’re talking about and start from there.”
Be kind. Sadly, reviewers don’t always write tactfully. As a writer, Lew has himself weathered unkind comments—and become a better reviewer in the process. “It’s a reminder to be civil, kind, and positive.” Harriman uses a litmus test of three questions when considering submitting his comments: ‘Is it true? Is it kind? Is it necessary?’”
Peer Review: Not Just for Science
Peer review gives the technical community the equivalent of a factory’s “Inspected by #8” sticker. Other fields that also use peer review to ensure quality include philosophy journals and scholarly journals on topics as diverse as law, art, and ethics.
Even those outside the research community use some form of peer review. Figure skating championships may be judged by former skaters and coaches. Artists help judge art contests. One may even think of Supreme Court rulings as one of the nation’s most trusted forms of peer review.