Optimizing Automated Essay Scoring with Lightweight Large Language Models and Validated Rubrics

Authors

  • Prayitno Politeknik Negeri Semarang https://orcid.org/0000-0002-4794-2957
  • Fahima Choirun Nabila Politeknik Negeri Semarang
  • Mohammad Khambali Politeknik Negeri Semarang
  • Afandi Nur Aziz Thohari Politeknik Negeri Semarang
  • Karisma Trinanda Putra Universitas Muhammadiyah Yogyakarta
  • Viqi Ardaniah Universitas Airlangga
  • Jurianto Universitas Airlangga

DOI:

https://doi.org/10.29207/resti.v10i2.7012

Keywords:

automated essay scoring, english as a foreign language essay, large language model, prompting method

Abstract

Manual grading of English as a Foreign Language (EFL) essays often leads to inconsistent scores among educators, despite the use of rubrics. While traditional Automated Essay Scoring (AES) systems offer speed, they often fail due to high computational cost, reliance on extensive datasets, and an inability to capture holistic writing qualities such as creativity and humanistic expression. This study addresses these issues by introducing AESCORE, a novel, lightweight, and cost-effective AES framework. Our methodology centers on integrating validated rubric criteria (identified via VOSviewer analysis) with open-source Large Language Models (LLMs), specifically emphasizing a human-centered approach. We evaluated AESCORE across 100 EFL essays using several prompting techniques, including few-shot and multi-trait specialization. The system achieved its most robust performance and high scoring consistency (Quadratic Weighted Kappa QWK = 0.6660) using the DeepSeek-R1 8B LLM with few-shot prompting. AESCORE represents a significant contribution by demonstrating that sophisticated, pedagogically-aligned writing assessment and generative feedback can be achieved with accessible AI, offering a reliable alternative for improving productive writing skills in higher education.

Downloads

Download data is not yet available.

Author Biography

  • Prayitno, Politeknik Negeri Semarang

    Department of Electrical Engineering

References

M. Faseeh et al., “Hybrid Approach to Automated Essay Scoring: Integrating Deep Learning Embeddings with Handcrafted Linguistic Features for Improved Accuracy,” Mathematics, vol. 12, no. 21, 2024, https://doi.org/10.3390/math12213416

S. Mumford and D. Atay, “Teachers’ perspectives on the causes of rater discrepancy in an English for Academic Purposes context,” Assess. Writ., vol. 48, p. 100527, Apr. 2021, https://doi.org/10.1016/j.asw.2021.100527

N. B. Uzun, D. Alici, and M. Aktas, “Reliability of the Analytic Rubric and Checklist for the Assessment of Story Writing Skills: G and Decision Study in Generalizability Theory,” Eur. J. Educ. Res., vol. 8, no. 1, pp. 169–180, Jan. 2019, https://doi.org/10.12973/eu-jer.8.1.169

R. Conijn, P. Kahr, and C. Snijders, “The Effects of Explanations in Automated Essay Scoring Systems on Student Trust and Motivation,” J. Learn. Anal., vol. 10, no. 1, pp. 37–53, Mar. 2023, https://doi.org/10.18608/jla.2023.7801

M. Beseiso, O. A. Alzubi, and H. Rashaideh, “A novel automated essay scoring approach for reliable higher educational assessments,” J. Comput. High. Educ., vol. 33, no. 3, pp. 727–746, Dec. 2021, https://doi.org/10.1007/s12528-021-09283-1

A. V. Y. Lee, A. C. Luco, and S. C. Tan, “A Human-Centric Automated Essay Scoring and Feedback System for the Development of Ethical Reasoning,” Educ. Technol. Soc., vol. 26, no. 1, pp. 147–159, 2023, https://doi.org/10.30191/ETS.202301_26(1).0011

X. Li, L. Qu, M. Tan, and Y. Jia, “Automated essay scoring based on the enhanced chimp optimization algorithm-back propagation (ENChOA-BP) and K-means,” Multimed. Tools Appl., vol. 1–32, no. 2024, 2024, https://doi.org/10.1007/s11042-024-20313-9

X. Tang, H. Chen, D. Lin, and K. Li, “Harnessing LLMs for multi-dimensional writing assessment: Reliability and alignment with human judgments,” Heliyon, vol. 10, no. 14, 2024, https://doi.org/10.1016/j.heliyon.2024.e34262

Y. Song, Q. Zhu, H. Wang, and Q. Zheng, “Automated Essay Scoring and Revising Based on Open-Source Large Language Models,” IEEE Trans. Learn. Technol., vol. 17, pp. 1880–1890, 2024, https://doi.org/10.1109/TLT.2024.3396873

M. Uto and M. Okano, “Learning Automated Essay Scoring Models Using Item-Response-Theory-Based Scores to Decrease Effects of Rater Biases,” IEEE Trans. Learn. Technol., vol. 14, no. 6, pp. 763–776, Dec. 2021, https://doi.org/10.1109/TLT.2022.3145352

J. Ruan et al., “Applying Large Language Models to Power Systems: Potential Security Threats,” 2023, arXiv. https://doi.org/10.48550/ARXIV.2311.13361

A. Kundu and D. Barbosa, “Are Large Language Models Good Essay Graders?,” 2024, arXiv. https://doi.org/10.48550/ARXIV.2409.13120

S. Kukreja, T. Kumar, A. Purohit, A. Dasgupta, and D. Guha, “A Literature Survey on Open Source Large Language Models,” in Proceedings of the 2024 7th International Conference on Computers in Management and Business, Singapore Singapore: ACM, Jan. 2024, pp. 133–143. https://doi.org/10.1145/3647782.3647803

S. Joshi, M. S. Khan, A. Dafe, K. Singh, V. Zope, and T. Jhamtani, “Fine Tuning LLMs for Low Resource Languages,” in 2024 5th International Conference on Image Processing and Capsule Networks (ICIPCN), Dhulikhel, Nepal: IEEE, Jul. 2024, pp. 511–519. https://doi.org/10.1109/ICIPCN63822.2024.00090

W. Mansour, S. Albatarni, S. Eltanbouly, and T. Elsayed, “Can Large Language Models Automatically Score Proficiency of Written Essays?,” Apr. 16, 2024, arXiv: arXiv:2403.06149. https://doi.org/10.48550/arXiv.2403.06149

R. Zhang et al., “LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention,” Sep. 18, 2024, arXiv: arXiv:2303.16199. https://doi.org/10.48550/arXiv.2303.16199

H. Yoo, J. Han, S.-Y. Ahn, and A. Oh, “DREsS: Dataset for Rubric-based Essay Scoring on EFL Writing,” 2024, arXiv. https://doi.org/10.48550/ARXIV.2402.16733

J. Wilson and R. D. Roscoe, “Automated Writing Evaluation and Feedback: Multiple Metrics of Efficacy,” J. Educ. Comput. Res., vol. 58, no. 1, pp. 87–125, Mar. 2020, https://doi.org/10.1177/0735633119830764

Y. Karatay and L. Karatay, “Automated writing evaluation use in second language classrooms: A research synthesis,” System, vol. 123, 2024, https://doi.org/10.1016/j.system.2024.103332

Y.-S. Lee, “Analysis of Small Large Language Models(LLMs),” Int. J. Adv. Smart Converg., vol. 13, no. 4, pp. 155–160, Dec. 2024, https://doi.org/10.7236/IJASC.2024.13.4.155

S. Cahyawijaya, H. Lovenia, and P. Fung, “LLMs Are Few-Shot In-Context Low-Resource Language Learners,” 2024, arXiv. https://doi.org/10.48550/ARXIV.2403.16512

D. Ramesh and S. K. Sanampudi, “An automated essay scoring systems: a systematic literature review,” Artif. Intell. Rev., vol. 55, no. 3, pp. 2495–2527, Mar. 2022, https://doi.org/10.1007/s10462-021-10068-2

J. Fleckenstein, L. W. Liebenow, and J. Meyer, “Automated feedback and writing: a multi-level meta-analysis of effects on students’ performance,” Front. Artif. Intell., vol. 6, 2023, https://doi.org/10.3389/frai.2023.1162454

A. Alsharif, “Artificial Intelligence and the Future of Assessment: Opportunities for Scalable, Fair, and Real-Time Evaluation,” Libyan J. Educ. Res. E-Learn. LJERE, pp. 42–52, Feb. 2025, https://doi.org/10.65417/ljere.v1i1.5

M. Stahl, L. Biermann, A. Nehring, and H. Wachsmuth, “Exploring LLM Prompting Strategies for Joint Essay Scoring and Feedback Generation,” 2024, arXiv. https://doi.org/10.48550/ARXIV.2404.15845

Q. Wang and J. M. Gayed, “Effectiveness of large language models in automated evaluation of argumentative essays: finetuning vs. zero-shot prompting,” Comput. Assist. Lang. Learn., pp. 1–29, Jul. 2024, https://doi.org/10.1080/09588221.2024.2371395

L. Yoshida, “The Impact of Example Selection in Few-Shot Prompting on Automated Essay Scoring Using GPT Models,” in Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners, Doctoral Consortium and Blue Sky, vol. 2150, A. M. Olney, I.-A. Chounta, Z. Liu, O. C. Santos, and I. I. Bittencourt, Eds., in Communications in Computer and Information Science, vol. 2150. , Cham: Springer Nature Switzerland, 2024, pp. 61–73. https://doi.org/10.1007/978-3-031-64315-6_5

G.-G. Lee, E. Latif, X. Wu, N. Liu, and X. Zhai, “Applying large language models and chain-of-thought for automatic scoring,” Comput. Educ. Artif. Intell., vol. 6, p. 100213, Jun. 2024, https://doi.org/10.1016/j.caeai.2024.100213

T. Shibata and Y. Miyamura, “LCES: Zero-shot Automated Essay Scoring via Pairwise Comparisons Using Large Language Models,” 2025, arXiv. https://doi.org/10.48550/ARXIV.2505.08498

Prayitno, “Automatic AES dataset and Code.” Politeknik Negeri Semarang, Jul. 28, 2025. Accessed: Jul. 28, 2025. [Online]. Available: https://zenodo.org/records/16536693

H.-C. Chen et al., “Week-Wise Student Performance Early Prediction in Virtual Learning Environment Using a Deep Explainable Artificial Intelligence,” Appl. Sci., vol. 12, no. 4, p. 1885, Feb. 2022, https://doi.org/10.3390/app12041885

K. Putra et al., “Optimizing the Performance of AI Model for Non-Invasive Continuous Glucose Monitoring: Hyperparameter Tuning and Random Oversampling Approach,” JOIV Int. J. Inform. Vis., vol. 8, no. 2, p. 991, May 2024, https://doi.org/10.62527/joiv.8.2.2047

K. T. Putra et al., “Federated Compressed Learning Edge Computing Framework with Ensuring Data Privacy for PM2.5 Prediction in Smart City Sensing Applications,” Sensors, vol. 21, no. 13, p. 4586, Jul. 2021, https://doi.org/10.3390/s21134586

Prayitno et al., “A Systematic Review of Federated Learning in the Healthcare Area: From the Perspective of Data Properties and Applications,” Appl. Sci., vol. 11, no. 23, Art. no. 23, Nov. 2021, https://doi.org/10.3390/app112311191

D. Purbohadi and J. Santoso, “Child-friendly e-learning for artificial intelligence education in Indonesia: conceptual design,” IAES Int. J. Artif. Intell., vol. 14, no. 4, pp. 2622–2633, 2025, https://doi.org/10.11591/ijai.v14.i4.pp2622-2633

Sunardi, Prayitno, B. P. Kamiel, A. D. Saputri, Z. H. Muizza, and A. Yobioktabera, “Smart Harvest: Web-Integrated Ripeness Detection for Apples with CNN Algorithm,” Ingénierie Systèmes Inf., vol. 29, no. 6, pp. 2181–2190, Dec. 2024, https://doi.org/10.18280/isi.290608

Downloads

Published

2026-04-22

Issue

Section

Computer Science Applications

How to Cite

Prayitno, Fahima Choirun Nabila, Mohammad Khambali, Afandi Nur Aziz Thohari, Karisma Trinanda Putra, Viqi Ardaniah, & Jurianto. (2026). Optimizing Automated Essay Scoring with Lightweight Large Language Models and Validated Rubrics. Jurnal RESTI (Rekayasa Sistem Dan Teknologi Informasi), 10(2), 366-374. https://doi.org/10.29207/resti.v10i2.7012