Lior Pachter has an interesting observation
which goes back to an old article of Richard Guy extracting four major issues in interpreting data
- Superficial similarities spawn spurious statements.
- Capricious coincidences cause careless conjectures.
- Early exceptions eclipse eventual essentials.
- Initial irregularities inhibit incisive intuition.
Unfortunately this seems to describe the way we think and even worse – this is what the science system promotes: the spectacular, the unexpected, the fascinating news.
To continue his story, what is the lifetime of the spurious idea? In many instances effects are declining rapidly for example in intelligence research. It took me some time to find the first paper that I remember – it was in 2001 that John & Despina wrote that the results of the first study correlate only modestly with subsequent research on the same association. This was confirmed in 2005
Of 49 highly cited original clinical research studies, 45 claimed that the intervention was effective. Of these, 7 (16%) were contradicted by subsequent studies, 7 others (16%) had found effects that were stronger than those of subsequent studies, 20 (44%) were replicated, and 11 (24%) remained largely unchallenged.
A scandal? The list of failed studies is long, including all areas of biomedicine already back in 2015.