Scientific ChatGPT Generated Texts Can be Detected with 99% Accuracy

An AI-detection tool has been developed by scientists that can detect scientific text generated by AI with a 99% accuracy rate, targeting platforms like ChatGPT. The detection tool, which is uniquely designed to distinguish AI from human writing using a smaller dataset and human insight, narrows its focus to the specific scientific writing found in peer-reviewed journals, offering more accuracy than general-purpose detectors.

A new tool that can identify scientific text produced by ChatGPT, an artificial intelligence text generator, with 99% accuracy is developed by Heather Desaire, a chemist at the University of Kansas who applies machine learning to biomedical studies.

The effectiveness of her AI-detection technique was confirmed by a study published in the peer-reviewed journal Cell Reports Physical Science, which also contains enough source code for others to use. Defending scientific integrity requires the use of accurate AI-detection tools, as stated by Desaire, the Keith D. Wilner Chair in Chemistry at KU.

She pointed out that ChatGPT and other AI text generators are capable of falsifying facts, which can be detrimental to academic science publications by polluting the literature with plausible and sounding falsehoods. Automated methods cannot automatically identify such fictitious AI nonsense when it is used to populate genuine scientific facts with falsifiable AI nonsense that sounds perfectly plausible.

According to her, the effectiveness of her detection method depends on narrowing the scope of scientific writing under scrutiny to the type typically found in peer-reviewed journals, which improves accuracy over existing AI-detection tools such as the RoBERTa detector.

Desaire has made her team’s code for AI-detecting fully accessible to researchers who want to expand it. She hopes that others will come to the realization that AI and AI detection are within reach of those who don’t typically think of themselves as computer programmers.

People without a degree in computer science have a lot of potential to get involved in AI, even if they don’t. The authors on our manuscript didn’t have a computer science degree. I’d like to see people who are interested in AI understand that the barriers to creating useful products, like ours, are relatively low. With a little knowledge and some creativity, many people can contribute to this field.

Reference: “Distinguishing academic science writing from humans or ChatGPT with over 99% accuracy using off-the-shelf machine learning tools” by Heather Desaire, Aleesa E. Chua, Madeline Isom, Romana Jarosova and David Hua, 7 June 2023, Cell Reports Physical Science.