All posts by admin

Software

When AI results cannot be generalized

There is a new Science paper that shows

A central promise of artificial intelligence (AI) in healthcare is that large datasets can be mined to predict and identify the best course of care for future patients. … Chekroud et al. showed that machine learning models routinely achieve perfect performance in one dataset even when that dataset is a large international multisite clinical trial … However, when that exact model was tested in truly independent clinical trials, performance fell to chance levels.

This study predicted antipsychotic medication effects for schizophrenia – admittedly not a trivial task due to high individual variability (as there are no extensive pharmacogenetics studies behind). But why did it completely fail? The authors highlight two major points in the introduction and detail three in the discussion

  • models may overfit the data by fitting the random noise of one particular dataset rather than a true signal
  • poor model transportability is expected due to patients, providers, or implementation characteristics that vary across trials
  • in particular patient groups that are too different across trials while this heterogeneity is not covered in the model
  • missing outcomes and covariates like psychosocial information and social determinants of health were not recorded in all studies
  • patient outcomes may be too context-dependent where trials may have subtly important differences in recruiting procedures, inclusion criteria and/or treatment protocols

So are we left now without any clue?

I remember another example of Gigerenzer in “Click” showing misclassification of chest X rays due to different devices (mobile or stationary) which associates with more or less serious cases (page 128 refers to Zech et al.). So we need to know the relevant co-factors first.

There is even a first understanding of the black box data shuffling in the neuronal net. Using LRP (Layer-wise Relevance Propagation) the recognition by weighting the characteristics of the input data can already be visualized as a heatmap.

 

CC-BY-NC Science Surf , accessed 20.08.2026
Noteworthy

Data voids and search engines

An interesting Nature editorial reporting a recent study

A study in Nature last month highlights a previously underappreciated aspect of this phenomenon: the existence of data voids, information spaces that lack evidence, into which people searching to check the accuracy of controversial topics can easily fall…
Clearly, copying terms from inaccurate news stories into a search engine reinforces misinformation, making it a poor method for verifying accuracy…
Google does not manually remove content, or de-rank a search result; nor does it moderate or edit content, in the way that social-media sites and publishers do.

So what could be done?

There's also a body of literature on improving media literacy - including suggestions on more, or better education on discriminating between different sources in search results.

Sure increasing media literacy at the consumer site would be helpful. But letting Google earn all that money without any further curation efforts? The original study found

Here, across five experiments, we present consistent evidence that online search to evaluate the truthfulness of false news articles actually increases the probability of believing them.

So why not putting out red flags? Or de-rank search results?

fake screen shot

 

 

CC-BY-NC Science Surf , accessed 20.08.2026
Noteworthy

Das Ende der Bachelorarbeit

ist wohl schon eingeleitet zumindest bei der Betriebswirtschaft in Prag, Zitat

Texte, die mit Künstlicher Intelligenz verfasst wurden sind kaum von menschlichen zu unterscheiden. Eine Prüfung sei für Unis deshalb nur sehr schwer möglich, sagt Dekan Hnilica. “Wir haben andere Teile unseres Studiums, in denen die Studierenden ihre Lernergebnisse oder erwarteten Lernergebnisse nachweisen können. Daher ist die Bachelorarbeit überflüssig.”

 

CC-BY-NC Science Surf , accessed 20.08.2026
Noteworthy, Software

Scientific integrity is not a weapon

Bill Ackman is threadening Harvard faculty on X

I expect that in the not too distant future AI will target every paper and not only a suspicious table or an image found by chance. Nevertheless using this now as a weapon is immoral and at high risk of false accusations. And , it may even be prosecuted as criminal defamation.

 

Feb 11, 2025

Unfortunately scientific integrity is being used again as personal weapon. Stefan Weber is making a business from right wing clients to verify doctoral theses. Without doubt, he has excellent technical skills (or at least a Turnitin account) but also completely lost all sense of proportion and direction. See

Back to SPIEGEL yesterday, translated

In some cases, his accusations turned out to be unfounded or less serious than he portrayed them. That's why he is viewed more critically in Austria. … Until the Föderl-Schmid case, none of this had harmed him much. But for those he accused, it was a different story. Even if the allegations came to nothing, their reputation was tarnished

 

CC-BY-NC Science Surf , accessed 20.08.2026
Philosophy

PubPeer – an introduction for Science Journalists

5 tips for using PubPeer to investigate scientific research errors and misconduct

  • Install the PubPeer browser extension
  • Don't publish a news story simply to point out people have made negative comments
  • Verify all claims made on PubPeer
  • Use caution when describing the likelihood that researcher misconduct happened
  • No matter how small a role a researcher plays in your story, check PubPeer before including them

As an introductory text I can recommend a 5 year old blog entry introducing PubPeer

Battle lines are being drawn on the internet, between the scientific establishment and volunteer vigilantes trying to impose their own vision of the scientific process through “post-publication peer review”.

On one side is the cream of the scientific aristocracy: a professor with a meteoric career trajectory at Imperial College London, one of the best universities in the world, and the top academic publisher, Nature Publishing Group. On the other side: a few anonymous malcontents carping on an obscure web site called PubPeer (welcome to our site!).

 

CC-BY-NC Science Surf , accessed 20.08.2026
Software

The missing sleepwatcher manpage


sleepwatcher(8) System Manager's Manual sleepwatcher(8)

NAME
sleepwatcher - daemon to monitor sleep, wakeup and idleness of a Mac

SYNOPSIS
sleepwatcher [-n] [-v] [-V] [-d] [-g] [-f configfile] [-p pidfile]
[-a[allowsleepcommand]] [-c cantsleepcommand]
[-s sleepcommand] [-w wakeupcommand] [-D displaydimcommand]
[-E displayundimcommand] [-S displaysleepcommand]
[-W displaywakeupcommand] [-t timeout -i idlecommand
[-R idleresumecommand]] [-b break -r resumecommand]
[-P plugcommand] [-U unplugcommand]

DESCRIPTION
sleepwatcher is a program that monitors sleep, wakeup and idleness of a Mac. It can be used to execute a Unix command when the Mac or the display of the Mac goes to sleep mode or wakes up, after a given time without user interaction or when the user resumes activity after a break or when the power supply of a Mac notebook is attached or detached. It also can send the Mac to sleep mode or retrieve the time since last user activity.
Continue reading The missing sleepwatcher manpage →

 

CC-BY-NC Science Surf , accessed 20.08.2026