Even linguistics experts are largely unable to spot the difference between writing created by an AI or human, say University of South Florida researchers.
Research just published in the ScienceDirect journal Research Methods in Applied Linguistics revealed that experts from the world’s top linguistic journals could not differentiate between AI- and human-generated abstracts more than 39 percent of the time.
“We thought if anybody is going to be able to identify human-produced writing, it should be people in linguistics who’ve spent their careers studying patterns in language and other aspects of human communication,” said Matthew Kessler, a scholar in the USF the Department of World Languages, working with Elliott Casal, assistant professor of applied linguistics at The University of Memphis.
Linquists mostly clueless
So Kessler and associates tasked 72 experts in linguistics with reviewing a variety of research abstracts to determine whether they were written by AI or humans. Each expert was asked to examine four writing samples. None correctly identified all four; 13 percent got them all wrong.
Based on this, Kessler and Casal concluded ChatGPT can write short genres just as well as most humans—if not better in some cases: AI typically does not make grammatical errors.
Longer texts: probably generated by humans
But for longer texts, AI has been known to hallucinate and make up content, making it easier to identify that it was generated by AI,” Kessler said.
Kessler hopes this study will lead to a bigger conversation to establish the necessary ethics and guidelines surrounding the use of AI in research and education.
Citation: Casal, J. E., & Kessler, M. (2023). Can linguists distinguish between ChatGPT/AI and human writing?: A study of research ethics and academic publishing. Research Methods in Applied Linguistics, 2(3), 100068. https://doi.org/10.1016/j.rmal.2023.100068 (open-access)