Abstract

The accelerated development of sophisticated large language models (LLMs) poses a serious and increasingly critical risk to academic integrity as it effortlessly generates high-quality scientific text that it is often barely differentiable to work by humans. Current detection systems, often based on statistical analysis or single-feature analysis, have severe shortcomings in generalizability between AI models and are not available to the broader academic community. In response to these issues, this study introduced MAGNet, a new Multimodal Graph Neural Network architecture that can be used to identify AI-made scientific text with high robustness and flexibility. Our model goes beyond conventional and unites and combines several feature modalities such as syntactic patterns, semantic relationships, and graph-based structural features into a single deep learning model. With such design, MAGNet is capable of finding weak, subtle artifacts and trends that are typical of AI generation that simpler statistical methods fail to detect. Trained on 5,000 scientific texts, a highly rigorously curated dataset, to guarantee a direct and fair comparison with modern studies, our model has an accuracy of 91.0% state-of-the-art, with a precision of 89.5% and a recall of 93.0%. Most importantly, MAGNet has shown a high level of performance in difficult situations, achieving the accuracies of 87.1% in cross-model generalization test and 84.5% in deliberately paraphrased documents, thus mentioning its high resilience to evasion behaviors. We have created and implemented a publicly accessible web application, to connect the gap between research and practice to provide the academic community with a powerful and user friendly interface to use this new advanced detection capability. Not only does our work set a new standard in performance in AI-text detection, but it also provides a highly valuable practical resource of educators, researchers, and publishers interested in protecting the integrity of scientific communication. The study concluded that introduced a new multimodal, graph-based neural network called MAGNet that determines AI-written scientific text. They showed that a combination of semantic, statistical, and structural analysis in the same deep learning system shows the highest performance, and the accuracy of the system is 91.0% on a large-scale, curated dataset.

Keywords

  • AI
  • Deep Learning
  • Natural language processing
  • academic integrity and multi-modal fusion

References

  1. 1 OpenAI, "GPT-4 Technical Report" 2023, arXiv:2303.08774.
  2. 2 J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, BERT: Pre-training Deep Bidirectional Transformers Language Understanding, in Proc. NAACL, 2019, pp. 4171-4186.
  3. 3 H. Touvron et al., "LLaMA: Open and Efficient Foundation Language Models," 2023, arXiv:2302.13971.
  4. 4 Anthropic, "Claude 3 Model Card," 2024. [Online]. Available: https://www.anthropic.com/news/claude-3-family
  5. 5 T. Brown et al., the article Language Models are Few-Shot Learners, in Adv. Neural Inf. Process. Syst., 2020, vol. 33, pp. 1877-1901.
  6. 6 S. Gehrmann, H. Strobeld, and A. M. Rush, GLTR: Statistical Detection and Visualization of AI-Generated Text, Proc. ACL, 2019, p. 1-6.
  7. 7 E. Frantar et al., Efficient Detection of LLM-Generated Text with Statistical Benchmarking, in Proc. IEEE Int. Conf. Data Mining (ICDM), 2021, pp. 432-441.
  8. 8 Solaiman et al., Release Strategies and the Social Impacts of Language Models, ArXiv preprint arXiv:1908.09203, 2019.
  9. 9 J. Kirchenbauer et al., A watermark to large language models, Proc. ICML, 2023, pp. 17061-17084.
  10. 10 V. Christlein et al., On the Unreliability of Perplexity to AI-Generated Text Detection, IEEE Trans. Inf. Forensics Security, vol. 18, pp. 3473-3486, 2023.
  11. 11 Y. Liu et al., 2019, arXiv:1907.11692 RoBERTa: A Robustly Optimized BERT Pretraining Approach.
  12. 12 P. In Proc. ICLR, 2021.
  13. 13 P. Velickovic et al., Graph Attention Networks, in Proc. ICLR, 2018.
  14. 14 A. Vaswani et al., Attention is All You Need, in Adv. Neural Inf. Process. Syst., 2017, vol. 30.
  15. 15 L. Wang et al., "A Comprehensive Analysis of AI Text Detection Tools," in Proc. IEEE BigData, 2023, pp. 1120-1129.
  16. 16 D. Lund et al., The threat of fake research generated by AI, Ethics Hum. Res., vol. 45, no. 4, p. 2-11, Jul. 2023.
  17. 17 N. Uppal et al., The Generality Problem: Analyzing Model Blindness in AI-Generated Text Detection, in Proc. NAACL, 2024, pp. 215-228.
  18. 18 Z. Wu et al., Graphics: A Comprehensive Survey on Graph Neural Networks, IEEE Trans. Neural Netw. Learn. Syst., vol. 32, no. 1, pp. 4-24, Jan. 2021.
  19. 19 T. Mikolov et al., "Efficient estimation of word representations in vector space," in Proc. ICLR, 2013.
  20. 20 J. Howard and S. Ruder, University language model Fine-tuning of text classifier, in Proc. ACL, 2018, pp. 328-339.
  21. 21 M. E. Peters et al., Deep Contextualized Word Representations, Proc. NAACL, 2018, p. 2227-2237.
  22. 22 A. Radford et al., "Generative Pre-Training to Understand Language Better," OpenAI, 2018.
  23. 23 A. Radford et al., “Language Models are Unsupervised Multitask Learners, OpenAI, 2019.
  24. 24 M. Z. Hossain et al., ACM Comput. Surv., vol. 55, no. 10, pp. 1-38, Oct. 2023.
  25. 25 U. of Cambridge, "AI and Academic Integrity: Policy and Practice," Cambridge Acad. Integr. Rev., vol. 4, no. 2, pp. 45-62, 2023.