How This Scientist Reviews a Paper.

Hello, Dear Readers!

I am feeling particularly less annoyed than in my last post. There have been a few posts written that thoroughly describe why the pig-chimp hybrid idea is NOT science: check out Donald Prothero's post, and this article by Henry Gee, which also includes what has to be the most disturbing image of a pig-primate merger that I have had the displeasure to see. (Thanks, Guardian. Now my subconscious has new fodder with which to replace the spiders that usually invade my nightmares.)

Nope, I decided that I can't be the only one who has to live with this image. You're welcome.

All of the criticisms of the idea are consistent, but the take-home for non-scientists should be this: the hybrid idea is not science because there is nothing that has been peer-reviewed and published. As I detailed in my last post, throwing information up on the Internet is not the same as publishing a peer-reviewed paper in a scientific journal. There is no credibility in such a website because there has been no fact-checking. That's what the peer-review process is: scientific fact and logic checking by other scientists. Plain and simple.

What exactly is involved with peer-reviewing a paper?

I can describe the process I go through when I review a paper that has been submitted for scientific publication. In fact, I'm reviewing an ichnology paper right now, so I can outline what I do and in what order I do it.

Note: I will not be divulging the authors, contents, or my comments of said paper. That behavior would be the weakest of weak tea. One of my specialties is ichnology, so it stands to reason that I will be asked to review ichnology papers. The first people who get to see the detailed comments and recommendations are the authors themselves and the editor of the scientific journal. Actually, the ONLY people who get to see my comments are the paper authors and the editors. Confidentiality in this part of the review process is important for maintaining objectivity. The only person I will have direct contact with will be the editor. Once the paper is published, its data and interpretations are then open to the scientific community as a whole for comment and critique, but only the finished product: even then my comments and recommendations remain confidential.

First, I have to actually agree to review a paper. I am given a choice whether I want to review, but am under no specific obligation to review a paper. I do not get paid for the time it takes to review papers (there is no financial reasons involved in accepting or rejecting a paper.) There is nothing in my job description that states "Thou shalt review thy colleague's papers!" While there is no specific obligation, the expectation is still there to review. Why? Well, for many reasons:
- I have accumulated knowledge and experience, and can make constructive comments that will hopefully improve the content and/or presentation a paper,
- Part of contributing to science is to make sure that shoddy science (i.e. interpretations that are not supported by the available data) are corrected and do not make it into official publications,
- Papers cannot be published in scientific journals unless they are reviewed. If people don't review papers, the process is stalled, and it slows down spreading the scientific love,
- I also publish papers, and someone in my field took the time to review and improve my papers. This is a science version of "paying it forward."

Second, I have to agree to the time limit for the review. These can range from a couple of weeks to a couple of months. The idea is that I will get the review completed in a timely enough fashion so that the paper can be published in a timely fashion.

OK. I've accepted the review request and the editor has sent the PDFs of all of the files that have been submitted by the authors. These include the text of the paper, all of the images and their captions, all of the data tables and their captions, and all of the supplementary information.

The first official step that I take is I read the abstract. Yup, just like reading a paper for my own research project, I read the abstract to see what sort of paper I will be reviewing. Is it an systematic paper, where new ichnotaxa will be named? Is it a first occurrence (or earliest occurrence, or youngest occurrence) of tracks an area? Is it a ichnofaunal paper? Each type of paper has its own specific rules for presenting that specific information, so I make a mental check list of what I want to see in such a paper. I also make a mental check list of what reasons I would accept for certain bits of information not being presented. Heck, I'm a field person, so I know that the realities of collecting data in the field will trump the ideals. I just want to see that the authors have made an effort to not only collect all the information that they need, but to account for information they wanted to collect but couldn't because the site was unexpectedly flooded, or a tornado hit, or it just wouldn't stop raining, or the site was vandalized, or they lost funding and couldn't continue the work....you get the idea.

If there are certain track types or ichnofossils mentioned in the abstract, I make note of those, because the next step for me is to actually read the key papers that have been previously published on those specific trace fossils. Oh yes, Dear Reader. This paper reviewer does her homework. I don't get to just sit back with a cup of tea (OK, the tea part I do get to do), whip out the Red Pen of Dashed Hopes and arbitrarily cross out paragraphs whilst yelling "Rubbish!"

My ideal Red Pen of Dashed Hopes would have feathers on it. Grading and reviewing papers is more fun when the cat that is physically sitting on said papers is entertained. Priorities. Image from froggooseandbear.blogspot.com

There is no way I can retain all of the relevant details of every ichnotaxon ever described in my tea-soaked noggin. I don't assume that I know (or can remember) everything. I read up on the original descriptions of the ichnotaxa in question, as well as the key interpretations that have been previously made about said ichnotaxa. I want this information fresh in my head going into the review. I also review any of the debates or alternate interpretations regarding some ichnotaxa and the evidence to support said arguments. I also make a mental check of my own stances on certain arguments. I've been around the outcrop a few times, so I have my own opinions on what interpretations the available data does and doesn't support.

My background reading is complete, but the materials are close at hand in case I need a refresher, or to make sure that the authors are interpreting previous works accurately, or are at least not stretching an interpretation too far. One item I always have at hand is the International Code of Zoological Nomenclature. Yup, ichnologists must follow the Code for the naming of new ichnospecies, ichnogenera, and ichnofamilies, just like any other taxonomist. If the paper I'm reviewing is naming, renaming, or reorganizing the systematics of a certain ichnogroup, it's my job as Reviewer to make sure the rules of the Code are followed.

I've gone through the paper and have highlighted areas that I find either need correcting (grammar, typos, etc.) or more work (examining alternate interpretations, more data, better figures, and in the name of the Flying Spaghetti Monster include a data table if there are measurements collected!) It's not just my job to say "This isn't good enough," or "NO!" [That "NO!" was an actual review comment received by a colleague.] Those are examples of general negative comments at don't actually help the author improve the paper. I have to qualify what kind of work is needed to get me from a "No!" to a "Yes!" If there are other papers that the authors need to use in certain sections, I list those. If I find the figures uninformative, I suggest what would need to change to make the figures useful. If there are alternate interpretations that need consideration, I state those. Review comments should always be constructive.

After my comments are all sorted out, I check the reference list. I go through each in-text citation and make sure that it occurs in the reference list. This is a long but necessary process. The reference list is a useful tool for people reading the paper. It contains all the sources of information used by the authors to arrive at their current interpretations. If the reference list is not complete or is full of errors, the reader of the finished product does not have all of the information.

Let's say you get your paper back, and it has a big ol' "NO!" written on one of the pages, or all of your great insights were met with harsh criticism by your peer reviewers. Yes, you might receive negative comments. Are you unpleasantly surprised? A bit pissed off? A little hurt? A lot hurt? You might be: you're also a human being.

I was teaching university-level science writing seminars, the inevitable writing assignment always created a lot of stress among the students, especially around the time when the first drafts were returned. I was fortunate in that, by the time I was teaching those writing seminars, I had already received a fairly damning review of one of my papers. I knew what it felt like to construct what I thought was a flawless piece of scientific brilliance only to have it crawl back from gaping maw of review, leaving a trail of smeared bloody red ink on the floor to mark its passing.

I was brutally honest with my students. I told them exactly what I thought and felt when receiving that review. I told them about all that personal gooshy feely stuff that us robotic scientists aren't supposed to experience. Then I told them how I proceeded from the raw emotion stage to the "making it better" stage.

I felt the feels, and then let them pass. After a couple of days, I picked up the review and looked at it again, this time with a critical eye. Well, darn it if my reviewers weren't quite correct in all of their comments and critiques! I found that two day cool-off period essential. I even implemented it for my students: they had to wait 48 hours before contacting me with questions about their papers. When they finally did contact me, I received less emotional responses and more constructive advice-seeking. At the end of the semester I even had a couple of students thank me for telling them the story of my first negative review and for making them wait a couple of days before coming to see me. The two days gave them time to see that my comments were there to help, not hurt.

How did my colleague deal with the "NO!" written across one of their pages? After setting it aside for a couple of days, they realized that something was striking the reviewer as "off" about the interpretations made in that section. They realized that perhaps they didn't provide enough background information to logically support the interpretations. Two pages of detailed explanation and support later, that "NO!"disappeared, and the interpretation was well supported. Even negative reviews can be helpful if you choose to see them in that light.

Time to switch from reviewing mode to paper-writing mode! Let's see if I can get one last paper submitted before the end of 2013.

SAS.

Strange Woman Abroad: Chongqing Ichnology Conference - The Lotus Tracksite

Well, Dear Readers, it's time to resume the saga of our 2012 Chongqing Ichnology Conference and related field trips. We visited the Yangqing National Geopark Tracksite (we made a return trip in September 2013, but that is a story for another time), and the Shandong area tracksites within the quarries. Now we were whisked away from the cool dry climate of north-central China to the southern coastal region of Chongqing. This would be the last field work we did before the conference on November 29.

In northeastern BC I've had the privilege of working on some spectacular paleontology sites with glamorous scenery that inspires one to run through alpine meadows singing. The Lotus Tracksite is one of those sites. Situated outside of the city of Chongqing, it gives off a sense of idyllic rural peacefulness.

"The hills are alive with the sound of footprints! Lalalalaaaahaaaa!" (This is why I am a scientist and not a singer.) Mountain scenery of the Lotus Stockade Tracksite.

For the next four days, the bus would drive us to the rural center, and then we would walk along a path that went over brook and field.

We followed the path to the farmhouse in the upper right corner of the photo.

Laura Pinuela and Martin Lockley.

Lovely terraced farms.

Babbling brook with sediment from a recent rain.

A completely different style of growing crops than I was used to seeing growing up in southern British Columbia farm country.

I was not brave enough to try one of the fresh spicy peppers seen on this pepper plant. I hang my head in shame. (No one else tried one either, so I am in good company!)

Our party reached the farmhouse, but that was not the end of the trek. Oh no: we had to transport ourselves (and our field gear) from the base of the cliff to a natural fissure using a cement stairway.

Daniel Marty and the resident farm ducks. These ducks spent a great deal of time free-ranging in the fields. They are also a possible modern analog for a webbed bird footprint from the Cretaceous, named Uhangrichnus.

Our brief reprieve at the farmhouse. Lida Xing (left), Martin Lockley (right).

Can you see the tracksite from here? If I were going to hide from raiders, this would be the place.

Martin actually counted the steps: there are approximately 900 steps in the staircase. This is where I experienced a slight ping of culture shock, because around noon and in the afternoons we would see elderly people slowly walking up and down the entire stairway. They were doing this for exercise. Coming from a culture where everything is in walking distance yet people drive less than a kilometer to get a bag of potato chips, this impressed me.

So many steps...

The Lotus Tracksite from the stairs. Almost there!

If it wasn't for the staircase, I could not see any easy egress or regress to the Stockade. It was a perfect defensive post against raiders.

As the Lotus Stockade is now a heritage tourism destination (thanks to the work of Lida Xing), there were visitor friendly additions to the actual site (including the stairs). Wooden boardwalks are installed so that people can view the tracks without stepping on them. As the derelicts that ichnologists are, the first thing we did was physically explore the track surface.

Approaching the tracksite.

Wooden barricades keep distracted paleontologists corralled.

The Lotus Stockade has a fascinating history. The people who took shelter there 700 years ago noticed the large ornithopod tracks and the large-scale dessication features. They arrived a very astute explanation for the presence of these odd shapes: the ornithopod tracks were described as preserved impressions of lotus blossoms. One of the fascinating parts of ichnology is discovering all of the cultural explanations for footprints, and this is one of my favorites. Lida did a wonderful artistic interpretation of the story, and it became one of the logos for the Ichnology Conference.

The interpretation of ornithopod tracks as preserved lotus impressions does not require a large leap in imagination. Check out the original image here.

Now it was time to get to work. As these tracks are currently the subject of in-progress and in-press papers, I won't show too many of the details that are going to occur in said papers. However, there are two surfaces that we had to document in three days: the upper ornithopod track layer, and the lower bird layer.

The upper ornithopod surface, viewable through a plexiglass floor. Us delinquent ichnologists had special permission to crawl all over the surface.

We split into two teams. Rich, Martin, Laura, and I took the lead on the bird layer, while Daniel, Matteo, Hendrik, and Julien worked on the ornithopod layer. The two teams had slightly different procedures, but both ended with two large plastic tracings that matched up between the layers beautifully. Lida bounced between Team Ornithopod and Team Bird (my affectionate names for the groups - we were all one happy ichnology team) making sure that we had everything we needed for a three-day session of mass data collection. He also made sure we were fed: a constant supply of snacks were made available.

Here's how Team Bird tackled a several square meter surface that contains over 200 footprints (218 according to my field notes.) First, we found a very distinct footprint with a clear digit III preserved. The end of digit III was the anchor point for the next step. Second, we established a 1 meter by 1 meter grid system along the surface. We established the grid using fairly old-school techniques: a compass, a meter stick, and chalk.

Next, we physically gridded the entire track-bearing surface. We went through a great deal of chalk! We saved our welder's soapstone pens for the next step of physically outlining each and every footprint we could see on the surface. Establishing the physical grid took about half a day (we checked, double-checked, and triple-checked for repeatability) while the outlining of the prints took about another half of a day. It is a long process, but it makes the data collection all the more simple.

Next, we labeled each footprint according to the grid square in which it occurred (e.g. track B10-1, A2-7, etc.) Each print was photographed from multiple angles for future digitizing work, as well as each individual grid square. After the tracks were labeled and individually photographed, physical measurements were collected (I rarely collect data from photographs if I have the original specimens at hand.)

Rich McCrea (foreground) of Team Bird photographing gridded squares while Martin Lockley paints latex on a specimen for replication. Team Ornithopod is in the background.

Team Ornithopod tracing the upper surface.

We were all quite amazed when we found out that we didn't have to hike all the way back down to farmhouse for lunch. A hot lunch was brought up to us! Salted pork, cabbage and hot peppers, rice, and stewed pumpkin! You can see from the photos that we are all in our cold weather gear, and a hot lunch was most welcome.

A hot lunch at the Lotus Tracksite.

Team Bird spent the bulk of their time physically measuring prints and taking photos. It wasn't until November 28 that we joined the tracing party.

The final step: tracing the surface onto plastic sheets. Not only did we trace on the prints on the plastic map, but we traced on the corners of the grid squares for reference.

Other researchers and conference attendees arrived during our three day session at the Lotus Track, which allowed for more eyes on the surface.

Team Bird was finished the documentation of the lower surface late in the afternoon of the 28th. It's an odd feeling to be finished documenting a surface. I always feel as though I should be doing something else, but we had done everything we could do in the time frame available. We were done. Team Ornithopod continued into the evening, but they too were finished on the 28th. Now all we had to do was to put our finishing touches on our presentations and switch to conference mode! The conference was fascinating, and I am pleased to say that my presentation on the multivariate analysis of Mesozoic bird footprints was very well received.

The traced plastic maps, all bundled up and ready to head to the lab for interpretation.

My patience was well rewarded! I had finally seen and documented bird footprints! Not only that, but I had seen bird footprints in an area steeped with history. My co-authors and I are working on getting the paper that analyzes these prints ready for submission. Stay tuned!

That's it for now!

SW out.

Public Service Science Announcement (PScA): Posted Online Does Not Mean Scientifically Published.

Good Morning, Dear Readers!

I'm feeling annoyed with the Internet right now. Perhaps I woke up cranky. Perhaps I haven't had my usual dose of Earl Grey tea. Or perhaps it's all these tweets I've been seeing over the past couple of weeks. I don't feel right posting a link or showing a screen capture of just one tweet as an example, because it seems that my Twitter feed is filled with retweets from multiple people sharing this news article. I wouldn't want to seem as though I was singling out one re-Tweeter.

The subject of said article: humans originated from a pig-chimpanzee hybridization.

"Holy Flying Spaghetti Monster! Which journal is this published in?" was my reaction when I first saw the links.

I may be incorrect, but there does not appear to be any scientific journal in which this research appears. The documentation for this idea comes from the researcher's own website.

My reaction when from "Holy Cthulhu!" to "I'll wait for the paper."

Science literacy and atrophied critical thinking skills are not new topics. We have the cringe-worthy "It's just a theory" phrase when someone wants to discredit an idea without themselves having any knowledge or data to support said discrediting. We have several cryptozoology websites whose contributors have boarded the Circular Reasoning Merry-Go-Round and present only data that supports their claims (Bigfoot/Sasquatch is my favorite example because of the strong ichnology influence.)

Topical to this is my all-time favorite, the Pacific Tree Octopus, a seemingly well-researched website that decries the decline of an endangered species, but was really a social science experiment to test the online academic critical thinking skills of students. Turns out that we as a society are a bunch of suckers when it comes to information presented online.

This is a definite PR problem when it comes to science publication. We (as scientists) call for open-access to published research while simultaneously seeming to denounce those that are sharing their own work for the world to see. The big difference between open-access of published data and these "open science" websites is that the published data has been officially published. This is not a smart-ass comment. When the results of a research project are published by a scientific journal, it means there is a history of critical work that went into getting those results to a certain scientific standard. This is called the peer-review process. First (there are variations of this process), the editor of the journal to which the paper was submitted would send the paper to two or three experts in the field of that particular research. These would be experts who were not directly involved with the research.

Second, the experts would go over the results with a critical eye. When I review papers, these are the questions I ask myself (and I assume that researchers reviewing my papers ask themselves):
- Were the data collected carefully? Were enough data collected?
- Does the subject matter (e.g. a fossil footprint) show to me what the researcher says that it shows?
- What are alternative interpretations that I can come up with before I reach the interpretations and conclusions section of the paper?
- Did the researcher address all the possible alternative interpretations? In the case of identifying a fossil, did the author examine all the possible identities of said fossil before settling on an identification?
- Did the researcher thoroughly check all of the previous work and interpretations that have been done on the subject?
- Are the interpretations and the conclusions in proportion to the data presented? In other words, are the authors stretching an interpretation over too little data?

Third, the experts would write up their comments and recommendations for the paper. Their recommendation options run from "Accept with no changes" to "Accept with minor/major changes" to "Accept with major changes and re-review" or "Reject/Not acceptable for publication." In my experience, the information and/or interpretations presented in a paper have to be pretty shoddy to warrant a "Reject." My philosophy with reviewing papers is to help the researchers present the most sound interpretation based on the data available. A rejected paper is not meant to be a personal attack: it simply means that the researcher did not convince their fellow researchers of their interpretations. It means they need to go back to the drawing board (or the lab, or the field, etc.) and do more work.

Fourth, the editor for the journal reads these recommendations, adds some of their own, and then notifies the researcher about the decision. If the paper is accepted with any type of revision recommended, the researcher must address those recommendations before sending in the corrected copy (or provide a darn good reason why they are not applying said recommendations.) The version of the paper that is originally submitted is the equivalent of a first draft of an essay, and the revised version is the second draft, ideally with all the corrections made. The peer-review process can be long, complicated, and sometimes arduous, but it is the most thorough way the scientific community has of fact-checking research.

Here's the peer-review process for posting your unpublished work online. First, do your work. Second, acquire internet access and design a website. Third, post said work. [Note: there is a big difference between posting unpublished research and posting about research that has already been published.]

This is why I always lectured my students in any writing classes I taught on the pitfalls of relying solely on the Internet for researching scientific writing assignments. There is no peer-review of the Internet. If I want to create a website that states Tyrannosaurus rex is a 40 meter long fish, I can do that. I can make it look official. I can present realistic looking data. Sure, I would be open to the mocking of my peers, but that website would remain up for as long as I wanted it to, and students everywhere could write about the wonders of an aquatic T. rex.

The fact that an image of an aquatic T. rex exists does make me less cranky. Image source: http://inveeous.blogspot.ca/

A less hyperbolic example of this is the discussion around the website ReptileEvolution.com: it is presented in such a way that the layperson or student would not be able to distinguish its information from peer-reviewed sources. Darren Naish at Tetrapod Zoology does a thorough examination of the site: it's a lengthy read, but worth every word. It is critical without being personal. Another example is seazoria.com: despite being told by countless paleontologists that the specimens in question are in fact glacial erratics with no bones in them whatsoever, the person who made the website insists that they represent the remains of a frozen "Sea Dragon."

This is why websites like the pig-chimp hypothesis, ReptileEvolution.com, and seazoria.com annoy me. Regardless of the intent of the authors of such "open science" websites, they do take advantage of a society that is hungry for scientific information but has atrophied critical thinking skills. The authors of these sites are already convinced of their ideas, but have not convinced their research peers. What are almost never presented on these sites are the rebuttals to these ideas, which only serves to make the ideas seem accepted by the scientific community.


Back to the chimp-pig topic. I don't think that I can do a better dissection than PZ Meyers on Pharyngula (who does poke fun, but also presents thorough counters), but from what I have read of the Macroevolution website, there is one misperception that I know for a fact is not accurate, and has the effect of making the lay-person reader more accepting of the hypothesis. [Note: I have to hat-tip the Macroevolution site here. They do not say this is a theory, but a hypothesis. The science-reporting world is the culprit for attaching the ever-misused "theory" moniker to the stories.]

This misconception is the non-acceptance of hybridization in tetrapods by the scientific community. At least in the avian world, there are several examples of hybridization that are not only accepted, but are the focus of scientific research. My favorite examples come from Dr. Darren Irwin's website on his lab's research at the University of British Columbia Department of Zoology (complete with links to published papers).

"You'll do." MacGillivray's Warbler (Geothlypis tolmiei) to the Mourning Warbler (Geothlypis philadelphia) during breeding season, producing the "MacMourning Warbler" above. Image source.

Not only do some bird species within the same genus interbreed, but they also produce fertile offspring. This is why there is such consternation among conservationalists regarding the Spotted Owl (Strix occidentalis): those darn endangered owls are known to get freaky with a closely related competitor species, the Barred Owl (Strix varia), which makes their conservation even more frustrating.

"Owl just have to settle for a Barred Owl" said the punny Spotted Owl during breeding season. Image source.

While this topic does give me an excuse to post bird (and specifically owl) images, there is a point: hybridization is not a verboten topic in biology. Hybrids can happen between closely related species. In other words, hybridization is not unheard of between species whose genetic lines have had only a short (few million years) amount of time to accumulate differences. As PZ Meyers details, approximately 80 million years worth of accumulated genetic differences is a different hybridization hurdle to overcome. This is just the one example that stuck in my mind. Other scientists with different research foci will likely find others.

Why this Public Service Science Announcement (PScA)? Unfortunate for the pig-chimp hypothesis and other sites like it, it has received a great deal of attention before receiving the "SCIENCE" stamp. For this idea to break into actual science, it has to go past a compilation of literature review and comparisons to a peer-reviewed published paper. All of the supporting data have to be critically viewed by other geneticists and evolutionary biologists, specifically those that have familiarity with the evolutionary history of the subjects in question. Both the argument and the data have to be convincing.

This is a PScA for students. If you plan to use an online source when doing research for your school projects (or even for your academic work if you are a grad student), please check with your teaching assistant or your professor. Chances are they have a list of sources that contain peer-reviewed information that will not lead you astray, and can also warn you about sites that seem legit but actually do not contain peer-reviewed information.

Always remember: just because the nice website offers you the intellectual version of candy doesn't mean that you should get into the vehicle with it.

Shaman out.