Measuring AI Writing on arXiv: Where Detection Breaks Down

The Statistical Ghost in the Machine

A recent analysis by the Unslop team took a hard look at a question haunting academic and engineering circles: can we actually measure how much of arXiv is written by AI? The source analysis didn't use a complex neural classifier. They used a brutally simple statistical proxy—tracking the frequency of specific "marker words" that LLMs overuse compared to human baseline writing.

The headline finding was stark. By analyzing a massive corpus of arXiv abstracts, they found a dramatic hockey-stick inflection point in late 2022, aligning perfectly with the public release of ChatGPT. The frequency of words like "delve," "showcasing," and "underscores" exploded. It’s not that humans never used these words; it’s that the rate of usage shifted from a flat, steady-state background noise to a vertical climb that correlates almost too perfectly with the mass availability of GPT-3.5 and GPT-4.

For engineers, this is a masterclass in using simple heuristics to solve a seemingly fuzzy problem. They didn't need to train a 7B parameter classifier; they needed a robust, interpretable, and computationally cheap signal. This approach sidesteps the black-box nature of deep learning detectors and gives a clear, auditable metric.

The Architecture of a Statistical Detector

To understand how this works, visualize the data pipeline. It’s not a complex neural net; it’s a high-throughput text processing system.

The beauty lies in the simplicity. The system doesn't try to "understand" the text. It counts. By bucketing these counts by month and institution, patterns emerge that are far more reliable than any single document’s classification score.

Why 'Delve' Became a Smoking Gun

If you’ve spent any time prompting LLMs for polished prose, you’ve seen the tics. "Delve" is the poster child. It’s a perfectly fine English verb that human academics use sparingly. But in the RLHF (Reinforcement Learning from Human Feedback) tuning process, models were heavily rewarded for sounding "thorough" and "exploratory." The token "delve" became a high-probability path to satisfy that reward model.

The analysis revealed that prior to 2022, the baseline usage of "delve" in arXiv abstracts was a gentle, almost flat line. Post-ChatGPT, the rate didn't just increase; it went exponential. We’re seeing the fingerprints of the reward model, not necessarily the base model.

This is critical for engineers building on top of LLMs. When you fine-tune or prompt a model, you aren’t just getting generic text; you’re getting text that has been optimized to game a specific reward function. These marker words are the artifacts of that gaming process. Other words in the cluster include:

• underscores
• showcasing
• pivotal
• comprehensive

These aren't jargon. They are stylistic filler that LLMs use to pad out an authoritative tone. The analysis found that first-author papers from certain regions showed a massively higher adoption rate of these markers, revealing not just that AI was used, but hinting at how it was used—likely as a translation or polishing tool.

The Engineering Trap: False Positives and Bias

Here’s where the measurement breaks down, and where every engineer needs to pay attention. The moment you turn a correlation into a rigid rule, you ship a broken product.

The Unslop analysis is honest about its limitations. A spike in "delve" is a proxy, not a conviction. The paper explicitly notes the danger of false positives, particularly for non-native English speakers. Many ESL (English as a Second Language) writers naturally acquire vocabulary through reading academic papers—the very papers now being generated by LLMs. They aren't using ChatGPT; they are learning English from a corpus contaminated by ChatGPT.

A rigid detector that flags "delve" as AI-generated will disproportionately penalize:

1. Non-native speakers who learned formal English via academic literature.
2. Disciplines that naturally use that vocabulary (e.g., philosophy or literary criticism).
3. Time-shifted baselines. What was anomalous in 2023 is now standard human vocabulary in 2025 because humans adapt to the linguistic environment.

This is the central engineering failure of AI text detectors. They are static classifiers in a dynamic, adversarial system. As models evolve from GPT-4 to Claude 3.5 Sonnet and beyond, the "tells" change. "Delve" might be suppressed in the next RLHF pass, replaced by a new tic. A detector built on marker words has a shelf life measured in months, not years.

How to Run This Analysis Yourself

You don’t need a PhD in NLP to replicate this kind of measurement. If you want to analyze AI adoption in a specific corpus—your company’s internal docs, customer support tickets, or open-source repos—you can do it with a few Python scripts.

The pipeline is straightforward:

1. Corpus Acquisition: For arXiv, use the S3 bulk access or the OAI-PMH API. For internal data, dump your text into a columnar format.
2. Preprocessing: Strip formatting, tokenize by word, and lowercase everything. Don’t over-clean; you want to preserve the exact token choices.
3. Frequency Counting: Count occurrences of your target marker words per document, normalized by document length (e.g., per 1000 words).
4. Temporal Plotting: Group by month. Use a rolling average to smooth noise.
5. Change Point Detection: Don’t just eyeball it. Use a simple algorithm like PELT (Pruned Exact Linear Time) to mathematically detect the shift in mean frequency.

import numpy as np
import ruptures as rpt

# Assume 'monthly_frequency' is a numpy array of normalized counts
model = rpt.Pelt(model="rbf").fit(monthly_frequency)
change_points = model.predict(pen=10)
print(f"Detected shift at indices: {change_points}")

This gives you a statistically defensible inflection point. You can then map that index back to a calendar date. This method is far more robust than setting an arbitrary threshold (e.g., ">2 'delves' per abstract = AI").

If you’re interested in building systems that handle complex, unstructured data like this, you might find the approach outlined in our guide on Building a SQL Analyst Agent That Queries Your Postgres Database Using Gemini useful for structuring your analysis pipelines.

The FDE Reality Check: Why Rigid Detectors Are a Losing Game

For Field Development Engineers (FDEs) and solutions architects, this research isn’t just an academic curiosity—it’s a warning about product strategy. I’ve seen too many startups try to sell "AI detection" as a feature. It’s a trap.

Selling a detector that uses brittle heuristics like marker words is a churn-generating machine. Here’s why:

1. The False Positive Backlash: When a detector falsely flags a customer’s legitimate work as AI-generated, the damage to trust is catastrophic. You aren’t just wrong; you’re accusing the user of fraud. This is a high-stakes classification problem where precision must be astronomically high, and that’s mathematically impossible with statistical proxies alone.
2. The Adversarial Loop: The moment you release a detector, you train the adversary. Students and paper mills will simply add a post-processing step: "Rewrite this text to avoid the following words: delve, showcase, pivotal." Your detector is now bypassed by a one-line prompt.
3. Watermarking vs. Detection: The source notes that true cryptographic watermarking (altering the token sampling process at inference time) is the only theoretically sound solution. But this requires model-level access and cooperation from the provider. A post-hoc detector is always playing catch-up.

For FDEs embedding with customers, the playbook isn't to sell a magic detector. It’s to help customers build processes that account for AI usage. Instead of asking "Was this written by AI?", ask "Is this work correct, reproducible, and original?"

This aligns with the broader shift we’ve discussed in Agent Swarms and the New Model Economics: Why Routing to Smaller Models Wins. The future isn’t a single monolithic model or detector; it’s a swarm of smaller, specialized checks. One agent checks for logical consistency. Another checks for code execution reproducibility. A third checks for citation accuracy. A statistical marker-word check can be a signal in that swarm—a low-weight feature—but never the judge.

The real value for an FDE is helping a customer integrate these checks into their existing CI/CD or review pipelines. You’re not selling a verdict; you’re selling observability into a process. And that’s a much stickier, higher-value conversation, much like the strategies we outline in How AI-Native Startups Use FDEs to Win Complex Enterprise Deals and Reduce Churn.

FAQ

Q: Can I reliably detect if a single document is AI-written using these marker words? No. The analysis is statistical and works only across large populations. Using marker words to judge an individual paper will result in an unacceptable number of false positives, especially for non-native English speakers.

Q: Why don’t we just use a classifier like GPTZero? Classifiers trained on GPT-3.5 output often fail on GPT-4 or Claude text because the token distributions shift. They are fragile and require constant retraining. They also suffer from the same bias issues as marker-word analysis.

Q: How did the researchers know the baseline frequency before AI? They used arXiv data from before the release of ChatGPT (pre-November 2022) as a "clean" historical baseline. The assumption is that mass-market LLM usage in academic writing was negligible before this date.

Q: Is it unethical to use LLMs for polishing academic writing? The analysis doesn’t make ethical judgments. It simply measures the trend. The engineering challenge is that current tools blur the line between "polishing" (fixing grammar) and "generating" (writing original thoughts), making blanket policies difficult to enforce.