A Guide to Interpreting Journal Paper Reviews

Below is a guide on how to interpret the review forms that members of the review group complete when reading a paper. This guide is intended to aid those members of the group unfamiliar with our reviewing process, as well as help any external interested parties who may use our reviews to decide whether to read a paper or not.

Review Form Overview

The reviews that the group provides of papers are standardised. This is for two reasons:

1. The review group acts as a workshop for junior researchers within the HCW lab. As such, a standardised review acts as a prompt for what members should look for when understanding and interpreting academic papers.
2. To promote the comparison of papers, even when they are written from differing research perspectives. Academic papers are written for a diverse range of people, yet they all exude similar characteristics. These include a contribution that the paper is making to the field, a rigorous method or technical approach that can be repeated in other experiments and clarity of presentation to aid the reader. By comparing these shared characteristics, members of the review group can gage what makes a paper good and/or bad even when the paper is from a field they are not completely familiar with.

The review form can be split into 3 distinct areas:

1. Paper Description: This provides details of the paper such as its title and authors in addition to the name of the reviewer.
2. Generic Paper Qualities: This is a matrix whereby generic qualities of academic papers are assessed.
3. Paper Specific Qualities: These are a series of qualities that are specific to the paper being reviewed, such as the contribution that the paper is trying to make to the field.

Part 1: Paper Description

Part 1 of the form requires information about the paper being reviewed. These fields are self explanatory. We require the title of the paper so we know the paper that the review is for. We also require the publication date and place of publication so that if the paper is interesting, it can be easily found. The reviewer and review date are mainly for completeness of records.

Part 2: Generic Paper Qualities

Part 2 of the form assesses the generic qualities of the paper being reviewed. All academic papers, regardless of the field, possess certain characteristics. These include a contribution to the field that the paper claims to have made, a method that is generic and repeatable in future experimentations and a clear presentation of the work. We ask group members to interpret these three qualities on a scale of 1 – 7 and supply comments that justify their choice of rating. A detailed description of how to apply a rating to these three generic qualities can be found below.

Interpreting The Generic Qualities Matrix

	Poor (1)	Neutral (4)	Good (7)
Originality / Significance	The paper is of no interest to the community and contains no original or significant material.	The paper contains some interesting ideas that contribute to a small incremental step in the field.	The paper offers a very significant contribution to the field that is of great interest to the community.
Technical / Method	The paper has a weak method or is technically flawed. The reviewer has no confidence in the work or the results the are presented.	The method or technical approach used in the paper is sound but is limited in its applicability to other areas of research.	The method or technical approach used is rigorous, providing the reviewer with complete confidence in the results obtained. The approach is also general enough to be used in a variety of situations.
Clarity of Presentation	The presentation of the work in the paper is poor to the extent of confusing.	The presentation of the work is acceptable but needs improvement. There are some areas that are vague or difficult to interpret.	The work is presented in a clear and unambiguous manner.

These scores merely act as a guideline and we encourage reviewers to use the comments section to justify and explain the rating that they have given to the paper for each quality.

Part 3: Paper Specific Qualities

Part 3 of the form describes in detail features that are unique to the paper. The reviewer clearly states the contribution that the paper claims to have made to the field as well as any results that are presented. While all papers should have a contribution, it may not always be the case that the paper has results, especially if it is a preliminary investigation. The reviewer also provides a summary of their thoughts on the paper before providing an overall score for the work, again on a scale of 1 – 7.