Optimizing Automated Essay Scoring with Lightweight Large Language Models and Validated Rubrics
DOI:
https://doi.org/10.29207/resti.v10i2.7012Keywords:
automated essay scoring, english as a foreign language essay, large language model, prompting methodAbstract
Manual grading of English as a Foreign Language (EFL) essays often leads to inconsistent scores among educators, despite the use of rubrics. While traditional Automated Essay Scoring (AES) systems offer speed, they often fail due to high computational cost, reliance on extensive datasets, and an inability to capture holistic writing qualities such as creativity and humanistic expression. This study addresses these issues by introducing AESCORE, a novel, lightweight, and cost-effective AES framework. Our methodology centers on integrating validated rubric criteria (identified via VOSviewer analysis) with open-source Large Language Models (LLMs), specifically emphasizing a human-centered approach. We evaluated AESCORE across 100 EFL essays using several prompting techniques, including few-shot and multi-trait specialization. The system achieved its most robust performance and high scoring consistency (Quadratic Weighted Kappa QWK = 0.6660) using the DeepSeek-R1 8B LLM with few-shot prompting. AESCORE represents a significant contribution by demonstrating that sophisticated, pedagogically-aligned writing assessment and generative feedback can be achieved with accessible AI, offering a reliable alternative for improving productive writing skills in higher education.
Downloads
References
M. Faseeh et al., “Hybrid Approach to Automated Essay Scoring: Integrating Deep Learning Embeddings with Handcrafted Linguistic Features for Improved Accuracy,” Mathematics, vol. 12, no. 21, 2024, https://doi.org/10.3390/math12213416
S. Mumford and D. Atay, “Teachers’ perspectives on the causes of rater discrepancy in an English for Academic Purposes context,” Assess. Writ., vol. 48, p. 100527, Apr. 2021, https://doi.org/10.1016/j.asw.2021.100527
N. B. Uzun, D. Alici, and M. Aktas, “Reliability of the Analytic Rubric and Checklist for the Assessment of Story Writing Skills: G and Decision Study in Generalizability Theory,” Eur. J. Educ. Res., vol. 8, no. 1, pp. 169–180, Jan. 2019, https://doi.org/10.12973/eu-jer.8.1.169
R. Conijn, P. Kahr, and C. Snijders, “The Effects of Explanations in Automated Essay Scoring Systems on Student Trust and Motivation,” J. Learn. Anal., vol. 10, no. 1, pp. 37–53, Mar. 2023, https://doi.org/10.18608/jla.2023.7801
M. Beseiso, O. A. Alzubi, and H. Rashaideh, “A novel automated essay scoring approach for reliable higher educational assessments,” J. Comput. High. Educ., vol. 33, no. 3, pp. 727–746, Dec. 2021, https://doi.org/10.1007/s12528-021-09283-1
A. V. Y. Lee, A. C. Luco, and S. C. Tan, “A Human-Centric Automated Essay Scoring and Feedback System for the Development of Ethical Reasoning,” Educ. Technol. Soc., vol. 26, no. 1, pp. 147–159, 2023, https://doi.org/10.30191/ETS.202301_26(1).0011
X. Li, L. Qu, M. Tan, and Y. Jia, “Automated essay scoring based on the enhanced chimp optimization algorithm-back propagation (ENChOA-BP) and K-means,” Multimed. Tools Appl., vol. 1–32, no. 2024, 2024, https://doi.org/10.1007/s11042-024-20313-9
X. Tang, H. Chen, D. Lin, and K. Li, “Harnessing LLMs for multi-dimensional writing assessment: Reliability and alignment with human judgments,” Heliyon, vol. 10, no. 14, 2024, https://doi.org/10.1016/j.heliyon.2024.e34262
Y. Song, Q. Zhu, H. Wang, and Q. Zheng, “Automated Essay Scoring and Revising Based on Open-Source Large Language Models,” IEEE Trans. Learn. Technol., vol. 17, pp. 1880–1890, 2024, https://doi.org/10.1109/TLT.2024.3396873
M. Uto and M. Okano, “Learning Automated Essay Scoring Models Using Item-Response-Theory-Based Scores to Decrease Effects of Rater Biases,” IEEE Trans. Learn. Technol., vol. 14, no. 6, pp. 763–776, Dec. 2021, https://doi.org/10.1109/TLT.2022.3145352
J. Ruan et al., “Applying Large Language Models to Power Systems: Potential Security Threats,” 2023, arXiv. https://doi.org/10.48550/ARXIV.2311.13361
A. Kundu and D. Barbosa, “Are Large Language Models Good Essay Graders?,” 2024, arXiv. https://doi.org/10.48550/ARXIV.2409.13120
S. Kukreja, T. Kumar, A. Purohit, A. Dasgupta, and D. Guha, “A Literature Survey on Open Source Large Language Models,” in Proceedings of the 2024 7th International Conference on Computers in Management and Business, Singapore Singapore: ACM, Jan. 2024, pp. 133–143. https://doi.org/10.1145/3647782.3647803
S. Joshi, M. S. Khan, A. Dafe, K. Singh, V. Zope, and T. Jhamtani, “Fine Tuning LLMs for Low Resource Languages,” in 2024 5th International Conference on Image Processing and Capsule Networks (ICIPCN), Dhulikhel, Nepal: IEEE, Jul. 2024, pp. 511–519. https://doi.org/10.1109/ICIPCN63822.2024.00090
W. Mansour, S. Albatarni, S. Eltanbouly, and T. Elsayed, “Can Large Language Models Automatically Score Proficiency of Written Essays?,” Apr. 16, 2024, arXiv: arXiv:2403.06149. https://doi.org/10.48550/arXiv.2403.06149
R. Zhang et al., “LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention,” Sep. 18, 2024, arXiv: arXiv:2303.16199. https://doi.org/10.48550/arXiv.2303.16199
H. Yoo, J. Han, S.-Y. Ahn, and A. Oh, “DREsS: Dataset for Rubric-based Essay Scoring on EFL Writing,” 2024, arXiv. https://doi.org/10.48550/ARXIV.2402.16733
J. Wilson and R. D. Roscoe, “Automated Writing Evaluation and Feedback: Multiple Metrics of Efficacy,” J. Educ. Comput. Res., vol. 58, no. 1, pp. 87–125, Mar. 2020, https://doi.org/10.1177/0735633119830764
Y. Karatay and L. Karatay, “Automated writing evaluation use in second language classrooms: A research synthesis,” System, vol. 123, 2024, https://doi.org/10.1016/j.system.2024.103332
Y.-S. Lee, “Analysis of Small Large Language Models(LLMs),” Int. J. Adv. Smart Converg., vol. 13, no. 4, pp. 155–160, Dec. 2024, https://doi.org/10.7236/IJASC.2024.13.4.155
S. Cahyawijaya, H. Lovenia, and P. Fung, “LLMs Are Few-Shot In-Context Low-Resource Language Learners,” 2024, arXiv. https://doi.org/10.48550/ARXIV.2403.16512
D. Ramesh and S. K. Sanampudi, “An automated essay scoring systems: a systematic literature review,” Artif. Intell. Rev., vol. 55, no. 3, pp. 2495–2527, Mar. 2022, https://doi.org/10.1007/s10462-021-10068-2
J. Fleckenstein, L. W. Liebenow, and J. Meyer, “Automated feedback and writing: a multi-level meta-analysis of effects on students’ performance,” Front. Artif. Intell., vol. 6, 2023, https://doi.org/10.3389/frai.2023.1162454
A. Alsharif, “Artificial Intelligence and the Future of Assessment: Opportunities for Scalable, Fair, and Real-Time Evaluation,” Libyan J. Educ. Res. E-Learn. LJERE, pp. 42–52, Feb. 2025, https://doi.org/10.65417/ljere.v1i1.5
M. Stahl, L. Biermann, A. Nehring, and H. Wachsmuth, “Exploring LLM Prompting Strategies for Joint Essay Scoring and Feedback Generation,” 2024, arXiv. https://doi.org/10.48550/ARXIV.2404.15845
Q. Wang and J. M. Gayed, “Effectiveness of large language models in automated evaluation of argumentative essays: finetuning vs. zero-shot prompting,” Comput. Assist. Lang. Learn., pp. 1–29, Jul. 2024, https://doi.org/10.1080/09588221.2024.2371395
L. Yoshida, “The Impact of Example Selection in Few-Shot Prompting on Automated Essay Scoring Using GPT Models,” in Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners, Doctoral Consortium and Blue Sky, vol. 2150, A. M. Olney, I.-A. Chounta, Z. Liu, O. C. Santos, and I. I. Bittencourt, Eds., in Communications in Computer and Information Science, vol. 2150. , Cham: Springer Nature Switzerland, 2024, pp. 61–73. https://doi.org/10.1007/978-3-031-64315-6_5
G.-G. Lee, E. Latif, X. Wu, N. Liu, and X. Zhai, “Applying large language models and chain-of-thought for automatic scoring,” Comput. Educ. Artif. Intell., vol. 6, p. 100213, Jun. 2024, https://doi.org/10.1016/j.caeai.2024.100213
T. Shibata and Y. Miyamura, “LCES: Zero-shot Automated Essay Scoring via Pairwise Comparisons Using Large Language Models,” 2025, arXiv. https://doi.org/10.48550/ARXIV.2505.08498
Prayitno, “Automatic AES dataset and Code.” Politeknik Negeri Semarang, Jul. 28, 2025. Accessed: Jul. 28, 2025. [Online]. Available: https://zenodo.org/records/16536693
H.-C. Chen et al., “Week-Wise Student Performance Early Prediction in Virtual Learning Environment Using a Deep Explainable Artificial Intelligence,” Appl. Sci., vol. 12, no. 4, p. 1885, Feb. 2022, https://doi.org/10.3390/app12041885
K. Putra et al., “Optimizing the Performance of AI Model for Non-Invasive Continuous Glucose Monitoring: Hyperparameter Tuning and Random Oversampling Approach,” JOIV Int. J. Inform. Vis., vol. 8, no. 2, p. 991, May 2024, https://doi.org/10.62527/joiv.8.2.2047
K. T. Putra et al., “Federated Compressed Learning Edge Computing Framework with Ensuring Data Privacy for PM2.5 Prediction in Smart City Sensing Applications,” Sensors, vol. 21, no. 13, p. 4586, Jul. 2021, https://doi.org/10.3390/s21134586
Prayitno et al., “A Systematic Review of Federated Learning in the Healthcare Area: From the Perspective of Data Properties and Applications,” Appl. Sci., vol. 11, no. 23, Art. no. 23, Nov. 2021, https://doi.org/10.3390/app112311191
D. Purbohadi and J. Santoso, “Child-friendly e-learning for artificial intelligence education in Indonesia: conceptual design,” IAES Int. J. Artif. Intell., vol. 14, no. 4, pp. 2622–2633, 2025, https://doi.org/10.11591/ijai.v14.i4.pp2622-2633
Sunardi, Prayitno, B. P. Kamiel, A. D. Saputri, Z. H. Muizza, and A. Yobioktabera, “Smart Harvest: Web-Integrated Ripeness Detection for Apples with CNN Algorithm,” Ingénierie Systèmes Inf., vol. 29, no. 6, pp. 2181–2190, Dec. 2024, https://doi.org/10.18280/isi.290608
Downloads
Published
Issue
Section
License
Copyright in each article belongs to the author
- The author acknowledges that the RESTI Journal (System Engineering and Information Technology) is the first publisher to publish with a license Creative Commons Attribution 4.0 International License.
- Authors can enter writing separately, arrange the non-exclusive distribution of manuscripts that have been published in this journal into other versions (eg sent to the author's institutional repository, publication in a book, etc.), by acknowledging that the manuscript has been published for the first time in the RESTI (Rekayasa Sistem dan Teknologi Informasi) journal ;





