How Do We Fix Peer Review? (Because It’s Broken…)

If you’re an academic, scientist, business article contributor, or any other person that writes research-y things and attempts to have them published in reputable journals, you’ve probably been scarred by the peer review process.

It can feel like others are tearing your work to shreds, but the key is to stay confident.

Today, a scientific journal published an article I (and others) wrote about using machine learning to predict diagnostic test sensitivity in malaria. It took forever to get the article through peer review. From finding appropriate reviewers to the back-and-forth fighting of our work, the entire experience is exhausting (and fresh on my mind), so I decided I’d give my thoughts here.

Note: This isn’t a rant. On the contrary, this is more of a starting place for an open discussion. Painful peer review plagues us all — journals, editors, reviewers, and people wanting to publish their discoveries.

Peer Review is Necessary

Given the misinformation issues we’ve seen surrounding politics, the coronavirus, etc., we should all be all familiar with the term, “fake news”. With any reputable piece of work, you should be able to trace the sources (key concept: multiple sources) and find the supporting information to adjudicate the claims in the work.

In scientific research, this is no different. You should easily be able to see how the research was carried out, what the researchers used as references, etc. Given that there is a such a huge amount of pressure on researchers to publish papers, this can sometimes result in false information being claimed as facts. While it’s not always nefarious, incorrect scientific information being taken as fact has huge implications for a scientific discipline.

So, how is this mitigated? — A blinded jury trial by unknown peer volunteers that we shall call “peer review”.

Peer review is a necessary evil as it provides a barrier to halt junk from being published as real scientific discoveries. Thus, it enables us to read something from a reputable journal without having to question if it’s bogus or not.

For anyone who hasn’t experienced a peer review, let me explain the process:

  1. You submit your article to a journal and select the article topics or keyword tags (like “poison dart frogs” or “herpetology”).
  2. The journal’s pre-editorial team will review the article to determine if they are interested in the content being published in their journal.
  3. If they’re interested, they’ll send out your article to a number of peer reviewers that have expertise related to the article. Usually, the goal is to have 3 peer reviews completed on a given paper, though they’ll send out the article to far more reviewers as the success rate is really low.
  4. The peer reviewers will then skim, I mean “read”, your article and provide comments to the journal on everything from the article’s accuracy to the quality of the writing and even assess any conflicts of interest.
  5. If the reviews are positive enough, you’ll then get the chance to review their comments, respond to them, and revise the manuscript.
  6. Repeat steps 4 and 5 until you want to give up on academia altogether or until you eventually get enough positive endorsements from the peer reviewers (usually 2/3) at which point the editor can then accept the manuscript, whichever happens first. ✔️❌✔️
  7. Finally, work with the journal’s editor to make any final revisions and then they’ll eventually publish the paper.

Peer Review is Broken

This peer review process can take MONTHS. For the paper that I mentioned earlier, it sat in various peer review stages from March until September. At the end of the day, finding an appropriate set of reviewers is challenging and can take a long time to pull off. Then, making the reviewers happy with adequate revisions also takes time.

The overlapping parts of the Venn diagram get smaller and smaller the more niche your paper is.

It’s very hard to find reviewers that fit your exact discipline, especially if you’re an interdisciplinary scientist like me. I mean, how many researchers out there can review Plasmodium genetics AND machine learning methodologies AND have signed up with my journal of choice to be a reviewer AND is not someone I have worked with in the past?

Peer review often brings out the worst in people for some reason. I’ve experienced incredibly harsh comments over silly things like a figure being placed out of order. I think this is due to the fact that peer reviewers are volunteering to do reviews for a journal for no other reason other than it looks good on your CV. So, if you’re not getting paid to do it and you’re being pressured to get it done quickly, how can we really expect quality?

Since peer reviewing is a volunteer effort, it’s not uncommon to see someone halfway skim/read your article and make incorrect comments on it. This hurts your chances of getting published and wastes your time.

I’ve had this happen plenty of times where a reviewer leaves a harsh comment saying I didn’t explain something that I clearly covered 16 times in the paper. The reviewer just didn’t actually read it.

The reviews on my aforementioned paper were very positive, but a single overly negative reviewer can sink the whole ship. For example, my paper was rather interdisciplinary between machine learning, genomics, and vector-borne diseases (malaria). In the first round of reviews, 2 of the 3 responses were overwhelmingly positive, but the third person had woken up on the wrong side of the bed.

“This article plays into the hype that is machine learning.” — biased reviewer #3

He/she was incredibly biased against machine learning as a discipline, so I was doomed to never get a positive response from them. Sometimes, it’s a game of chance. Don’t be afraid to resubmit.

Peer Review is Fixable

Despite all its issues, I think peer review is a necessary step to promote scientific accuracy, reproducibility, ethics, and rigor.

Here are a few ideas that I have to fix peer review:

  1. Journals should do more of the work. With all the money that journals make from publication fees, reviewers should not have to worry about formatting issues and typos. This should be on the editorial team of the journal (who are getting paid). If I didn’t have to fix commas and figure captions in a review and could just focus on the content, my review process would be much shorter.
  2. Journals should compensate their damn reviewers. If journals actually paid their reviewers some nominal amount, there would be more people willing to review. Also, I think this would entice reviewers to perform a faster, better, more diligent job at reviewing. P.S.-I will happily accept Amazon gift cards to read a paper.
  3. Encourage non-academics to review papers. You don’t need to have a PhD in a given subject to provide meaningful discussion on a topic. In fact, having industry professionals provide their points of view would be really impactful in bridging the gap between what happens in the lab and in the real world. (Also, this helps spread out the burden by having more people to do the work.)
  4. Reformat peer review altogether. I hope to see other forms of peer review in the near future that allow for quicker publication, reduced opacity in the review process, and operate more like a discussion. Maybe we should try a Reddit-style review system where researchers can upvote or downvote an article into publication or removal?

I hope you’ve enjoyed my quick commentary on peer review, why it’s necessary, how it’s broken, and how we may go about fixing it.

Stay curious. CF

Cloud AI and genomics guy and aspiring polymath. I am a researcher in machine learning and bioinformatics and I sometimes write things here.