How can you tell whether a text was written by an AI?

Teachers, publishers and recruiters increasingly ask whether a text was written with an AI. There are 2 ways to answer: read the marks that models leave in their texts, or hand the text to an automatic detector. This explanation concerns texts; images call for other methods. The studies cited concern English: we have not found an equivalent survey for French.

Kobak and his co-authors counted the words in 15 million abstracts of biomedical articles published from 2010 to 2024 in the PubMed database. After the arrival of ChatGPT, at the end of 2022, some words became far more frequent than their trend would have led one to expect: in 2024, "delves" is 28 times more, "underscores" 13.8 times and "showcasing" 10.7 times. From this excess, the authors estimate that at least 13.5% of the abstracts of 2024 went through a language model, for writing or proofreading, and up to 40% in some subsets of journals or countries.

The contributors to English-language Wikipedia keep a list of the marks they meet in edits made with an AI. It cites words ("delve", "pivotal", "tapestry", "testament", "underscore"), the replacement of "is" by "serves as", parallelisms of the "not X, but Y" type, the repeated use of lists of 3 items, section titles with a capital on every word, and a final section on the "challenges" of the subject, always built on the same pattern. The list specifies that an isolated word proves nothing: it is the accumulation of these marks, in an edit made after 2022, that counts.

Assistants do not all leave the same marks. Sun and his co-authors trained a classifier on answers from 5 assistants, ChatGPT, Claude, Grok, Gemini and DeepSeek: it identifies the original assistant in 97.1 % of cases, where chance would give 20%. The marks lie in vocabulary and construction more than in form: after a paraphrase by another model, the classifier is still right in 91.4% of cases, and in 91.8% after translation into Chinese. It falls to 58.1% when the text is summarised.

A detector returns a score for a single text, which is harder than counting frequencies over millions. In 2023, Liang and his co-authors submitted to 7 widely used detectors texts written by humans: 91 TOEFL essays, the English exam for non-native speakers, collected from a Chinese forum, and 88 essays by US 8th-grade students, the equivalent of year 9. The detectors wrongly classed 61 % of the TOEFL essays as written by an AI, on average, against 5% of the US essays. 18 of the 91 TOEFL essays were flagged by all 7 detectors, and 89 by at least one. The authors explain this by simpler and more predictable vocabulary: after the vocabulary of these essays was enriched, the error rate falls to 12%.

Turnitin, publisher of a detector aimed at education, announced in June 2023 a false positive rate below 1% at document level, for documents containing at least 20% AI text, and, at sentence level, a probability of about 4% that a flagged sentence was in fact written by a human; 54% of the sentences wrongly flagged are next to a passage genuinely written by an AI. These rates are the publisher's, with no published external verification.

For us, at Projet Celsius, the marks of AI texts are real and measurable, but they are measured on corpora or between known assistants. A word or a phrase proves nothing about an individual text, because humans use them too, and those who write most simply are the most often wrongly accused. A detector score is not enough, on its own, to justify a sanction.