Unqualified Offerings

By Thoreau

Today I offer another inside view on the scientific community’s system of peer review. I do this to give an insider’s perspective on the goals of the system, and also to illustrate the limitations of the system (and to note that we scientists are very aware of those limitations).

I recently joined an editorial board for a new scientific journal that is trying to get off the ground. I was asked to review an article by a group developing a new image processing technique. I’m not a guru on image processing, I don’t have much formal training in the subject, and I don’t work on the specific problems being addressed by this technique. However, I’ve learned enough to do some work in the field (as a means to achieve certain goals in sub-diffraction optics). I know what sort of things should be addressed in a report on a new technique. And since I sometimes adopt other people’s tools, I know what a prospective user is looking for in a report on a new technique. Plus, this journal is new, so we don’t have a huge database of reviewers yet. I therefore felt a duty to render my best scientific judgement, rather than pass it on and wait for the optimal person to review it. It’s frequently the case that the exact perfect person won’t be called on, because there are only a few people working on any particular specific problem, and some will be busy while others have conflicts of interest.

Fortunately, this paper had some obvious strengths and weaknesses. I could clearly see what they were trying to do, the extent to which they had achieved that goal, and the important questions that they had left unanswered. So I recommended a revision that addressed a few key points. (The answer to your next question is no, I won’t tell you when the paper is published and point you to it. Reviews are anonymous.)

Things I want to convey to the layman: First, we scientists aren’t so different from anybody else in the workplace. Our formal training is often only distantly related to what we do. Most of what we do we learned on the job, and so sometimes we bring a practical but not necessarily detailed knowledge base to a task. It may very well be that a student who just took a formal class on certain topics in image processing could tell me all sorts of things that I didn’t know about the article that I just reviewed. However, the real issue in reviewing a paper is not always the details of each and every subsidiary point. Rather, the basic issue is whether the authors supported the points that they’re trying to make.

So that brings me to my second point: What scientists are looking for when we evaluate a paper is whether the paper clearly addresses 3 points:

1) What is the question or issue being studied in this work?

2) What are the methods being used, and are they described in a sufficiently detailed manner so that somebody else can replicate the work? (Remember that replication is the real gold standard of scientific knowledge. Until we have independent replication of a result, it’s suspect. Hell, even after independent replication we’re still skeptical.)

3) Does the data presented support the conclusions that the author is drawing? In the case of this work, I felt that the data was consistent with the conclusions, but not sufficient to give somebody confidence. So I asked for more data.

Now, you could ask whether it was wise for me to take the review rather than passing it on. Well, I would say it depends on how detailed and complicated of a claim the author is making. If the paper had made some very specific claims on some very complicated matters, I would have declined to review it, unless it was in a specialty that I have detailed knowledge of. However, the task that the authors were trying to accomplish was easy to understand, it was clear what the applications would be, and it was clear what criteria they were using to evaluate it. It was also clear what sort of hurdles they would have to overcome. FWIW, I’ve gotten referee reports from reviewers who didn’t seem to be experts on every point in my paper, but who could still home in on the key issues.

It’s entirely possible that there are some defects in the work that I overlooked. If so, then after it is put out there (and I expect that the authors will be able to address my concerns) more people can examine it, contact the authors, and send comments to the journal. If anything, it will be educational for all involved. Conversely, it may be that I’m missing some obvious point about their work, and so my concerns were unfounded. In that case, it will be easy for them to address my point, and some of the data that they add will be unnecessary from a technical perspective but nonetheless make for a paper that is more persuasive to the non-specialist looking for tools to apply in their research.

The simple fact of peer review is that it’s actually quite a modest hurdle. All you have to do is find one editor and a couple of reviewers who find the work plausible and well-executed. Once you’ve been on both sides of the process, you realize that it’s just a preliminary quality check, a first pass before it’s put out there for a wider audience. Some laymen seem to attribute too much significance to it, and other laymen seem to recoil against that misperception by concluding that peer review is too weak of a system. The truth is that it’s not supposed to be a stringent filter. It’s just supposed to be a first pass.

Anyway, just a few more (long and rambling) thoughts on the system we use to evaluate scientific publications.