{"id":22973,"date":"2023-12-09T11:31:03","date_gmt":"2023-12-09T09:31:03","guid":{"rendered":"https:\/\/www.wjst.de\/blog\/?p=22973"},"modified":"2023-12-10T17:11:49","modified_gmt":"2023-12-10T15:11:49","slug":"can-chatgpt-generate-a-rct-dataset-that-isnt-recognized-by-forensic-experts","status":"publish","type":"post","link":"https:\/\/www.wjst.de\/blog\/sciencesurf\/2023\/12\/can-chatgpt-generate-a-rct-dataset-that-isnt-recognized-by-forensic-experts\/","title":{"rendered":"Can ChatGPT generate a RCT dataset that isn’t recognized by forensic experts?"},"content":{"rendered":"

“Free synthetic data”? There are numerous Google ads selling synthetic aka fake data. How “good” are these datasets? Will they ever been used for scientific publications outside the AI field eg\u00a0 surgisphere-like<\/a>?<\/p>\n

There is a nice paper by Taloni,\u00a0 Scorcia and Giannaccare <\/a>that tackles the first question. Unfortunately a nature news commentary<\/a> by Miryam Naddaf is largely misleading when writing<\/p>\n

It will make it very easy for any researcher or group of researchers to create fake measurements on non-existent patients, fake answers to questionnaires or to generate a large data set on animal experiments.<\/p><\/blockquote>\n

Before I am going to provide the proof that it is much more complicated to generate any credible dataset, please read first the original paper<\/a> and the commentary<\/a> (BTW I do not expect<\/a> any high correlation between pre- and post vision acuity).<\/p>\n

As a reference here are the conditions how the data were generated. At this point I would like to thank the Italian colleagues<\/a> for giving me the dataset. DALK is <\/a>a shortcut for deep anterior lamellar keratoplasty and PK for penetrating keratoplasty.<\/p>\n

 <\/p>\n

\"\"<\/a>
click for details, unfortunately the quality of the supplemental PDF is low<\/figcaption><\/figure>\n

 <\/p>\n

Of course, there will be no entry of this RCT in any study registry. And there will be no other SOPs and no backup dataset if any commission would ever start an investigation. But can we just conclude from the data that they are being faked? Lets dive into the dataset.<\/p>\n

 <\/p>\n

\"\"<\/a>
Basic description<\/figcaption><\/figure>\n

 <\/p>\n

My first issue is not mentioned in the news commentary – there are NO missing values at all (#1). This is impossible in real life as every variable\u00a0 except of the study ID may include one or more missing values.<\/p>\n

 <\/p>\n

\"\"<\/a>
IDs<\/figcaption><\/figure>\n

 <\/p>\n

The “double eyes” are nonsense (#2).<\/p>\n

 <\/p>\n

\"\"<\/a>
Double IDs<\/figcaption><\/figure>\n

 <\/p>\n

We have a lot of gender\/sex discussions right now but in a surgery dataset from the last decade surnames and sex should match – so this is\u00a0 fatal flaw #3.<\/p>\n

Are there any any other missing or artificial associations?<\/p>\n

 <\/p>\n

\"\"<\/a>
Pearson Correlation Matrix<\/figcaption><\/figure>\n

 <\/p>\n

I am also plotting the predictive power score [ref<\/a>, ref<\/a>] as a simple Pearson correlation cannot detect non-linear bivariate relationships (without extensive preprocessing).<\/p>\n

 <\/p>\n

\"\"<\/a>
Predictive Power Score Matrix<\/figcaption><\/figure>\n

 <\/p>\n

Well, I am not an opthalmologist but the overall results are looking good to me. Not sure if there should be a correlation between cylinder size and acuity (BSCVA)? Shouldn’t females have slightly smaller cyclinder size? Is the cylinder size really somewhat related to left\/right eye? I find the data suspect but without a major flaw here. Of course a true clinical dataset would be very helpful for comparison.<\/p>\n

So let’s go to the distribution of the quantitative variables. There are quite nice Gaussian shapes except for the age distribution.<\/p>\n

 <\/p>\n

\"\"<\/a>
Probability distribution<\/figcaption><\/figure>\n

 <\/p>\n

So this is flaw #4 in particular as many patients are 17 years of age.\u00a0 Another flaw arises when looking at the date of of surgery.<\/p>\n

\"\"<\/a>
Surgery Dates<\/figcaption><\/figure>\n

 <\/p>\n

Surgery dates should be more or less equally distributed over the study period. Elective surgery is not expected to be done on weekends also, flaw #5 :-)<\/p>\n

Looking now at the qualitative variables and run this in subgroups by therapy. I can’t identify anything unusual as complications are correctly assigned.<\/p>\n

 <\/p>\n

\"\"<\/a>
Qualitative variables by outcome<\/figcaption><\/figure>\n

 <\/p>\n

So lets have a look into more advanced statistical procedures – Principal Component Analysis<\/a> and QQ plots<\/a>.<\/p>\n

 <\/p>\n

\"\"<\/a>
Principal Component Analysis<\/figcaption><\/figure>\n

 <\/p>\n

As expected PK and DALK have opposite factor loadings on the first component while I am not sure of the strong R\/L and M\/F loadings in PC2. Maybe some statisticians can jump in here and help?<\/p>\n

Here is also a QQ plot by surgical procedure.<\/p>\n

 <\/p>\n

\"\"<\/a>
QQ plots by outcome<\/figcaption><\/figure>\n

 <\/p>\n

Age is again unreliable as noted\u00a0 before while I now wonder also why BSCVA has only 5 levels? This is flaw #6.<\/p>\n

In a final step I look at the randomness of the second and third digit in two variables. Unfortunately,proportions are not equal according to a chi square test that \u00a0computes p-values by Monte Carlo simulation ( #7 ) as Benford’s law does not apply here.<\/p>\n

 <\/p>\n

\"\"<\/a>
Second and third digit distribution<\/figcaption><\/figure>\n

 <\/p>\n

Taken together,\u00a0 using a quick & dirty R script we have easily identified\u00a0 many flaws that clearly demarcate a fake dataset.<\/p>\n

 <\/p>\n

 <\/p>\n\n

 <\/p>\n

\n CC-BY-NC Science Surf , accessed 24.04.2026<\/span>\n <\/div>","protected":false},"excerpt":{"rendered":"

“Free synthetic data”? There are numerous Google ads selling synthetic aka fake data. How “good” are these datasets? Will they ever been used for scientific publications outside the AI field eg\u00a0 surgisphere-like? There is a nice paper by Taloni,\u00a0 Scorcia and Giannaccare that tackles the first question. Unfortunately a nature news commentary by Miryam Naddaf … Continue reading Can ChatGPT generate a RCT dataset that isn’t recognized by forensic experts?<\/span> →<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20,9],"tags":[4253,2554,3358,4034,4252,2550],"class_list":["post-22973","post","type-post","status-publish","format-standard","hentry","category-note-worthy","category-computer-software","tag-llm","tag-rct","tag-ai","tag-chatgpt","tag-eye-surgery","tag-fake"],"_links":{"self":[{"href":"https:\/\/www.wjst.de\/blog\/wp-json\/wp\/v2\/posts\/22973","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.wjst.de\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.wjst.de\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.wjst.de\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.wjst.de\/blog\/wp-json\/wp\/v2\/comments?post=22973"}],"version-history":[{"count":25,"href":"https:\/\/www.wjst.de\/blog\/wp-json\/wp\/v2\/posts\/22973\/revisions"}],"predecessor-version":[{"id":23038,"href":"https:\/\/www.wjst.de\/blog\/wp-json\/wp\/v2\/posts\/22973\/revisions\/23038"}],"wp:attachment":[{"href":"https:\/\/www.wjst.de\/blog\/wp-json\/wp\/v2\/media?parent=22973"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.wjst.de\/blog\/wp-json\/wp\/v2\/categories?post=22973"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.wjst.de\/blog\/wp-json\/wp\/v2\/tags?post=22973"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}