Detecting AI-Generated Text with Fine-Tuned RoBERTa: A Cross-Validated Study with Interpretability Analysis

Abstract

The rapid proliferation of large language models capable of generating fluent and contextually coherent text has made reliable authorship attribution both a practical and ethical necessity. Outputs from systems such as GPT-4, LLaMA 2, and Claude are increasingly indistinguishable from human-authored prose, yet current detection methodologies remain insufficient to meet this challenge. This study aims to develop and evaluate a fine-tuned RoBERTa-base binary classifier for distinguishing AI-generated text from human-authored text under stratified cross-validation conditions; assess the model's robustness to a known length-based confound through a controlled ablation study; and provide interpretable explanations of the classifier's decisions through dual token-level attribution using Gradient × Input Saliency and Attention Rollout. The model is trained and evaluated via stratified five-fold cross-validation on a balanced corpus of 6,069 samples (3,069 AI-generated and 3,000 human-authored) and benchmarked against four established baselines spanning classical, neural, and zero-shot paradigms. The classifier achieves a mean accuracy of 99.93%, a weighted F1-score of 0.9993, and a mean AUC-ROC of 1.0000 across all five folds. A length-balanced ablation produces an accuracy decline of only 0.11 percentage points, providing empirical evidence that the model has learned genuine semantic and stylistic distinctions rather than superficial length-based cues. Gradient × Input Saliency maps indicate that AI-generated texts are characterised by technically precise, domain-specific vocabulary, whereas human-authored texts are identified through discourse markers and rhetorically structured expression, with both attribution methods converging on the same high-saliency tokens.