How This Scientist Reviews a Paper.

Hello, Dear Readers!

I am feeling particularly less annoyed than in my last post. There have been a few posts written that thoroughly describe why the pig-chimp hybrid idea is NOT science: check out Donald Prothero's post, and this article by Henry Gee, which also includes what has to be the most disturbing image of a pig-primate merger that I have had the displeasure to see. (Thanks, Guardian. Now my subconscious has new fodder with which to replace the spiders that usually invade my nightmares.)

Nope, I decided that I can't be the only one who has to live with this image. You're welcome.

All of the criticisms of the idea are consistent, but the take-home for non-scientists should be this: the hybrid idea is not science because there is nothing that has been peer-reviewed and published. As I detailed in my last post, throwing information up on the Internet is not the same as publishing a peer-reviewed paper in a scientific journal. There is no credibility in such a website because there has been no fact-checking. That's what the peer-review process is: scientific fact and logic checking by other scientists. Plain and simple.

What exactly is involved with peer-reviewing a paper?

I can describe the process I go through when I review a paper that has been submitted for scientific publication. In fact, I'm reviewing an ichnology paper right now, so I can outline what I do and in what order I do it.

Note: I will not be divulging the authors, contents, or my comments of said paper. That behavior would be the weakest of weak tea. One of my specialties is ichnology, so it stands to reason that I will be asked to review ichnology papers. The first people who get to see the detailed comments and recommendations are the authors themselves and the editor of the scientific journal. Actually, the ONLY people who get to see my comments are the paper authors and the editors. Confidentiality in this part of the review process is important for maintaining objectivity. The only person I will have direct contact with will be the editor. Once the paper is published, its data and interpretations are then open to the scientific community as a whole for comment and critique, but only the finished product: even then my comments and recommendations remain confidential.

First, I have to actually agree to review a paper. I am given a choice whether I want to review, but am under no specific obligation to review a paper. I do not get paid for the time it takes to review papers (there is no financial reasons involved in accepting or rejecting a paper.) There is nothing in my job description that states "Thou shalt review thy colleague's papers!" While there is no specific obligation, the expectation is still there to review. Why? Well, for many reasons:
- I have accumulated knowledge and experience, and can make constructive comments that will hopefully improve the content and/or presentation a paper,
- Part of contributing to science is to make sure that shoddy science (i.e. interpretations that are not supported by the available data) are corrected and do not make it into official publications,
- Papers cannot be published in scientific journals unless they are reviewed. If people don't review papers, the process is stalled, and it slows down spreading the scientific love,
- I also publish papers, and someone in my field took the time to review and improve my papers. This is a science version of "paying it forward."

Second, I have to agree to the time limit for the review. These can range from a couple of weeks to a couple of months. The idea is that I will get the review completed in a timely enough fashion so that the paper can be published in a timely fashion.

OK. I've accepted the review request and the editor has sent the PDFs of all of the files that have been submitted by the authors. These include the text of the paper, all of the images and their captions, all of the data tables and their captions, and all of the supplementary information.

The first official step that I take is I read the abstract. Yup, just like reading a paper for my own research project, I read the abstract to see what sort of paper I will be reviewing. Is it an systematic paper, where new ichnotaxa will be named? Is it a first occurrence (or earliest occurrence, or youngest occurrence) of tracks an area? Is it a ichnofaunal paper? Each type of paper has its own specific rules for presenting that specific information, so I make a mental check list of what I want to see in such a paper. I also make a mental check list of what reasons I would accept for certain bits of information not being presented. Heck, I'm a field person, so I know that the realities of collecting data in the field will trump the ideals. I just want to see that the authors have made an effort to not only collect all the information that they need, but to account for information they wanted to collect but couldn't because the site was unexpectedly flooded, or a tornado hit, or it just wouldn't stop raining, or the site was vandalized, or they lost funding and couldn't continue the work....you get the idea.

If there are certain track types or ichnofossils mentioned in the abstract, I make note of those, because the next step for me is to actually read the key papers that have been previously published on those specific trace fossils. Oh yes, Dear Reader. This paper reviewer does her homework. I don't get to just sit back with a cup of tea (OK, the tea part I do get to do), whip out the Red Pen of Dashed Hopes and arbitrarily cross out paragraphs whilst yelling "Rubbish!"

My ideal Red Pen of Dashed Hopes would have feathers on it. Grading and reviewing papers is more fun when the cat that is physically sitting on said papers is entertained. Priorities. Image from froggooseandbear.blogspot.com

There is no way I can retain all of the relevant details of every ichnotaxon ever described in my tea-soaked noggin. I don't assume that I know (or can remember) everything. I read up on the original descriptions of the ichnotaxa in question, as well as the key interpretations that have been previously made about said ichnotaxa. I want this information fresh in my head going into the review. I also review any of the debates or alternate interpretations regarding some ichnotaxa and the evidence to support said arguments. I also make a mental check of my own stances on certain arguments. I've been around the outcrop a few times, so I have my own opinions on what interpretations the available data does and doesn't support.

My background reading is complete, but the materials are close at hand in case I need a refresher, or to make sure that the authors are interpreting previous works accurately, or are at least not stretching an interpretation too far. One item I always have at hand is the International Code of Zoological Nomenclature. Yup, ichnologists must follow the Code for the naming of new ichnospecies, ichnogenera, and ichnofamilies, just like any other taxonomist. If the paper I'm reviewing is naming, renaming, or reorganizing the systematics of a certain ichnogroup, it's my job as Reviewer to make sure the rules of the Code are followed.

I've gone through the paper and have highlighted areas that I find either need correcting (grammar, typos, etc.) or more work (examining alternate interpretations, more data, better figures, and in the name of the Flying Spaghetti Monster include a data table if there are measurements collected!) It's not just my job to say "This isn't good enough," or "NO!" [That "NO!" was an actual review comment received by a colleague.] Those are examples of general negative comments at don't actually help the author improve the paper. I have to qualify what kind of work is needed to get me from a "No!" to a "Yes!" If there are other papers that the authors need to use in certain sections, I list those. If I find the figures uninformative, I suggest what would need to change to make the figures useful. If there are alternate interpretations that need consideration, I state those. Review comments should always be constructive.

After my comments are all sorted out, I check the reference list. I go through each in-text citation and make sure that it occurs in the reference list. This is a long but necessary process. The reference list is a useful tool for people reading the paper. It contains all the sources of information used by the authors to arrive at their current interpretations. If the reference list is not complete or is full of errors, the reader of the finished product does not have all of the information.

Let's say you get your paper back, and it has a big ol' "NO!" written on one of the pages, or all of your great insights were met with harsh criticism by your peer reviewers. Yes, you might receive negative comments. Are you unpleasantly surprised? A bit pissed off? A little hurt? A lot hurt? You might be: you're also a human being.

I was teaching university-level science writing seminars, the inevitable writing assignment always created a lot of stress among the students, especially around the time when the first drafts were returned. I was fortunate in that, by the time I was teaching those writing seminars, I had already received a fairly damning review of one of my papers. I knew what it felt like to construct what I thought was a flawless piece of scientific brilliance only to have it crawl back from gaping maw of review, leaving a trail of smeared bloody red ink on the floor to mark its passing.

I was brutally honest with my students. I told them exactly what I thought and felt when receiving that review. I told them about all that personal gooshy feely stuff that us robotic scientists aren't supposed to experience. Then I told them how I proceeded from the raw emotion stage to the "making it better" stage.

I felt the feels, and then let them pass. After a couple of days, I picked up the review and looked at it again, this time with a critical eye. Well, darn it if my reviewers weren't quite correct in all of their comments and critiques! I found that two day cool-off period essential. I even implemented it for my students: they had to wait 48 hours before contacting me with questions about their papers. When they finally did contact me, I received less emotional responses and more constructive advice-seeking. At the end of the semester I even had a couple of students thank me for telling them the story of my first negative review and for making them wait a couple of days before coming to see me. The two days gave them time to see that my comments were there to help, not hurt.

How did my colleague deal with the "NO!" written across one of their pages? After setting it aside for a couple of days, they realized that something was striking the reviewer as "off" about the interpretations made in that section. They realized that perhaps they didn't provide enough background information to logically support the interpretations. Two pages of detailed explanation and support later, that "NO!"disappeared, and the interpretation was well supported. Even negative reviews can be helpful if you choose to see them in that light.

Time to switch from reviewing mode to paper-writing mode! Let's see if I can get one last paper submitted before the end of 2013.

SAS.