Tag Archives: Pubpeer

Peer review is science roulette

One of the best essays that I have read about current science.

Ricky Lanusse. How Peer Review Became Science’s Most Dangerous Illusion. https://medium.com/the-quantastic-journal/how-peer-review-became-sciences-most-dangerous-illusion-54cf13da517c

Peer review is far from a firewall. In most cases, it’s just a paper trail that may have even encouraged bad research. The system we’ve trusted to verify scientific truth is fundamentally unreliable — a lie detector that’s been lying to us all along.
Let’s be bold for a minute: If peer review worked, scientists would act like it mattered. They don’t. When a paper gets rejected, most researchers don’t tear it apart, revise it, rethink it. They just repackage and resubmit — often word-for-word — to  another journal. Same lottery ticket in a different draw mindset.  Peer review is science roulette.
Once the papers are in, the reviews disappear. Some journals publish them. Most shred them. No one knows what the reviewer said. No one cares. If peer review were actually a quality check, we’d treat those comments like gospel. That’s what I value about RealClimate [PubPeer, my addition]: it provides insights we don’t get to see in formal reviews. Their blog posts and discussions — none of which have been published behind paywalls in journals — often carry more weight than peer-reviewed science.

CC-BY-NC

Fellow Scientist: You must correct the record

Many PubPeer records highlight papers that are never corrected. Unfortunately there are many authors who never respond to comments in the hope that everything will be forgotten a few days later. Also Stanford’s Tessier-Lavigne hoped that time will let the dust settle while the Kirkland & Ellis Report by July 17,2023 clearly states

https://news.stanford.edu/stories/2022/12/statement-stanford-board-chair-jerry-yang
CC-BY-NC

PubPeer – an introduction for Science Journalists

5 tips for using PubPeer to investigate scientific research errors and misconduct

  • Install the PubPeer browser extension
  • Don’t publish a news story simply to point out people have made negative comments
  • Verify all claims made on PubPeer
  • Use caution when describing the likelihood that researcher misconduct happened
  • No matter how small a role a researcher plays in your story, check PubPeer before including them

As an introductory text I can recommend a 5 year old blog entry introducing PubPeer

Battle lines are being drawn on the internet, between the scientific establishment and volunteer vigilantes trying to impose their own vision of the scientific process through “post-publication peer review”.

On one side is the cream of the scientific aristocracy: a professor with a meteoric career trajectory at Imperial College London, one of the best universities in the world, and the top academic publisher, Nature Publishing Group. On the other side: a few anonymous malcontents carping on an obscure web site called PubPeer (welcome to our site!).

CC-BY-NC

Country analysis of PubPeer annotated articles

Just out of curiosity, after Scihub now an analysis of papers commented at the PubPeer website. Pubpeer is now also screened on a regular basis by Holden Thorp, the chief editor of Science…

Unfortunately I am loosing many records for incomplete or malformed addresses, while some preliminary conclusions can already be made when looking at my world map.

pubpeer.R grey indicates no data, black only a few, red numerous entries.

A further revision will need to include more addresses and also overall research output as a reference.

Continue reading Country analysis of PubPeer annotated articles

CC-BY-NC

ImageTwin

I confess that I worked together with the founder of ImageTwin some years ago, even encouraging him to found a company. I would have even been interested in a further collaboration but unfortunately the company has cut all ties.

Should we really pay now 25€ for testing a single PDF?

price list 2023

My proposal in 2020 was to build an academic community with ImageTwin’s  keypoint matching approach.  AI  analysis and image depository would be a nice along with more comprehensive reports than just drawing boxes around duplicated image areas.

A research paper  by new ImageTwin collaborators now finds

Duplicated images in research articles erode integrity and credibility of biomedical science. Forensic software is necessary to detect figures with inappropriately duplicated images. This analysis reveals a significant issue of inappropriate image duplication in our field.

Unfortunately the authors of this paper are missing the integrity nomenclature  flagging only images that are expected to look similar.

Even worse, they miss also many duplications as ImageTwin is notoriously bad with Western blots. Sadly, this paper erodes the credibility of forensic image analysis.

 

Oct 4, 2023

The story continues. Instead of working on a well defined data set and determining sensitivity, specificity, etc. of the ImageTwin approach, another preprint  (bioRxiv, Scholar) shows that

Toxicology Reports published 715 papers containing relevant images, and 115 of these papers contained inappropriate duplications (16%). Screening papers with the use of ImageTwin.ai increased the number of inappropriate duplications detected, with 41 of the 115 being missed during the manual screen and subsequently detected with the aid of the software.

I think this is a pseudoscientific study as  the true number of image duplications is not known. We cannot no more verify what ImageTwin does as it is behind a paywall contradicting basic scientific principles. The accompanying news report by Anil Oza  makes it even worse.

It is just wrong that the software is “working at two to three times David’s speed” – it is 20 times faster but  giving also numerous false positives.  It is wrong that “Patrick Starke is one of its developers”(Starke is a sales person not a developer). So at the end, the Oza news report is just a PR stunt as confirmed on Twitter on the next day

https://twitter.com/ImageTwinAI/status/1709842276929728610

Unfortunately ImageTwin has now been fallen back into the same league as Acuna et al. Not unexpected, Science Magazine has choosen Proofig for image testing, despite the nice groupshot of Starke and some other image sleuths.

CC-BY-NC

PubPeer Statistics

For a forthcoming article, I need some statistics to illustrate how PubPeer performs.  AFAIK know there is only one report from 2021 so  I had to try something by my own.

PubPeer submissions until 23 April 2022

Continue reading PubPeer Statistics

CC-BY-NC

PubPeer Pearls I

It’s always interesting if we can find a discussion under a PubPeer article with more than 3 comments. Elisabeth Bik collected some of these interesting #PubPeer Pearls at Twitter while I am starting a new collection here.

The longest thread that I remember is this one with 290 comments around a retracted article while my most appreciated PubPeer author Continue reading PubPeer Pearls I

CC-BY-NC

PubPeer should be merged into Pubmed (at some time point)

PubMed had an own comments feature “PubMed Commons” which had been shut down in 2018.

NIH announced it will be discontinuing the service — which allowed only signed comments from authors with papers indexed in PubMed, among other restrictions — after more than four years, due to a lack of interest.

But there is no lack of interest, if we look at the ever increasing rates at PubPeer – the counter today is 122.000.

The main  difference between PubMed Commons and PubPeer is the chance of submitting anonymous comments. While I also see a risk of unjustified accusations or online stalking, I believe that the current PubPeer coordinators handle this issue very well. We can post only issues that are obvious, directly visible or backed up by another source. Continue reading PubPeer should be merged into Pubmed (at some time point)

CC-BY-NC

Anonyme Wissenschaft

Gute Wissenschaft hängt nicht generell davon ab, ob sie mit einer bestimmten Person verknüpft werden kann – Sachverhalte sollte ja eigentlich objektiv reproduzierbar sein (auch wenn Wissenschaftsfunktionäre das anders sehen).

Interessant ist jedenfalls was  ein neues Journal nun erlaubt: die anonyme Veröffentlichung.

Cut to 180 years later, and philosophers are again asserting the right to publish under made-up names. But these philosophers, it seems, want to use pseudonyms to do the very thing Kierkegaard accused his contemporaries of doing: abstracting authors out of ethical reality … An “open access, peer-reviewed, interdisciplinary journal specifically created to promote free inquiry on controversial topics,” it will give authors the option to publish their work under a pseudonym “in order to protect themselves from threats to their careers or physical safety.”

In der Tat, nicht alle wissenschaftliche Erkenntnis kann unter gegebenen politischen Verhältnissen veröffentlicht werden. Es wird jedenfalls spannend werden

Pseudonymity is not an inherently bad thing. Apart from focusing the reader on the argument rather than the author, it can, in many cases, give a say to people who could otherwise not participate in public discourse.

CC-BY-NC

A cloud hangs over the paper in question

The title is taken from an essay of Dan Bolnick about current PubPeer practices

That said, there is some question about the proper procedure for answering these criticisms. Yes, PubPeer itself leaves room for comments (interestingly, journal editors like myself must pay money to reply to comments, even if to acknowledge them and state we are evaluating the issue). But, this process bypasses the journal that publishes the paper, and bypasses the normal scientific tradition of external review by experts in the field chosen by the journal editor for their knowledge and hopefully objectivity. For this reason, I want to really encourage people with substantial concerns about a paper (e.g., which may appreciably alter the results and conclusions), to submit formal “Comments” (different journals call these different things) to the journal.

I agree, letters or comments would be preferable. In practice, however, comments are largely ignored. I can provide numerous examples where nothing happened.

But, scientific traditions are fluid and we are in an era of increasing speed and openness: Preprint servers, open peer review, open data, et cetera. We therefore also recognize that PubPeer is an active tool in science conversations. The criticisms posted there can be valid identification of genuine problems that need to be evaluated formally and corrected. If valid well-justified and substantial concerns exist and are published on PubPeer, then the affected journal should respond.

Yea, yea. And are there caveats? Bolnick thinks

it is important that these not be used as a mechanism for pursuing personal vendettas. Excessive targeting of an author with multiple minor complaints can constitute a kind of harassment, and may be viewed as such by University Equity officers or equivalent. The anonymous nature of many PubPeer comments makes it easier for impacted authors to feel like (and, argue that) they are the target of personal vendettas and harassment. Second, the existence of PubPeer comments can cast a long shadow over a paper whether the comments are profound or minor. This shadow can affect an author’s career prospects (fellowship applications, job applications, etc) even before the matter is resolved and judged to be valid or not. The result can be inappropriate damage to an innocent authors’ career, which in turn may have grave consequences for mental health. Third, there is an established mechanism for voicing complaints about papers in science: Contact the author to request clarification, or contact the Editor, or submit a Comment.

Well, it depends. What is a minor complaint? One wrong label, two or three? Mixing up figures and data? Nirwana references? Intentional wrong statements? Ignorance of literature? If a comment is profound or minor can be decided by any PubPeer reader. Grave consequences for mental health of a fraudulent or careless author?

I believe that if there would be any working mechanism for voicing complaints about scientific integrity (or just minor corrections), PubPeer would not exist. Kudos Brandon Stell.

 

CC-BY-NC