Student-Generated User Story Quality: A Study on Practitioner and ChatGPT Evaluation

Authors

DOI:

https://doi.org/10.29207/resti.v9i5.6950

Keywords:

user stories, INVEST framework, practitioners, ChatGPT, quality evaluation

Abstract

Evaluating the quality of student-generated user stories is important in software engineering education, but only a limited number of industry practitioners can assist. The integration of generative AI can facilitate this process. To do so, the INVEST quality evaluation framework is widely recognized for assessing user story quality; however, prior research has not explored its use in conjunction with generative AI. This study investigated ChatGPT's ability to evaluate user stories using the INVEST framework. This study compares two ChatGPT-based evaluation approaches with those of experienced practitioners, focusing on student-generated user stories. Discrepancies between ChatGPT and practitioner evaluations were measured using Mean Absolute Deviation (MAD), Mean Squared Error (MSE), and Root Mean Squared Error (RMSE). Statistical significance was tested using the Mann-Whitney U Test. The results indicate that ChatGPT’s 1st approach yielded lower discrepancies than practitioner evaluations. Moreover, significance testing showed no statistically significant differences between the ChatGPT and practitioner results for the two INVEST criteria- Independent and Estimable. These findings suggest that the 1st approach can assist in the evaluation process, although practitioners must ensure comprehensive and accurate evaluations. ChatGPT can provide preliminary evaluations in educational contexts, enabling students to receive formative feedback and allowing educators to streamline evaluation processes. Although practitioner validation is still required, their role may shift toward verifying AI-generated results, thus reducing the overall workload and accelerating quality evaluation

Downloads

Download data is not yet available.

Author Biographies

  • Suhaila Mohd. Yasin, Universiti Tun Hussein Onn Malaysia

    Department of Software Engineering, Faculty of Computer Science and Information Technology

  • Dadang Syarif Sihabudin Sahid, Politeknik Caltex Riau

    Associate Professor in Information System, Politeknik Caltex Riau

References

A. Rasheed et al., “Requirement Engineering Challenges in Agile Software Development,” Math Probl Eng, vol. 2021, pp. 1–18, May 2021, doi: 10.1155/2021/6696695.

M. I. Zul, S. M. Yasin, and D. S. S. Sahid, “Exploring Requirement Engineering Challenges in Software Development: Insights from Global and Indonesian Landscape,” in 2024 4th International Conference on Electrical Engineering and Informatics (ICon EEI), IEEE, Oct. 2024, pp. 136–141. doi: 10.1109/IConEEI64414.2024.10748069.

A. Hendriana, T. Raharjo, and A. Nurfitriani, “Approaches in Determining User Story Quality through Requirement Elicitation : A Systematic Literature Review,” Indonesian Journal of Computer Science, vol. 12, no. 6, Jan. 2024, doi: 10.33022/ijcs.v12i6.3639.

G. B. Herwanto, G. Quirchmayr, and A. M. Tjoa, “Leveraging NLP Techniques for Privacy Requirements Engineering in User Stories,” IEEE Access, vol. 12, pp. 22167–22189, 2024, doi: 10.1109/ACCESS.2024.3364533.

L. Buglione and A. Abran, “Improving the user story Agile technique using the INVEST criteria,” in Proceedings - Joint Conference of the 23rd International Workshop on Software Measurement and the 8th International Conference on Software Process and Product Measurement, IWSM-MENSURA 2013, IEEE Computer Society, 2013, pp. 49–53. doi: 10.1109/IWSM-Mensura.2013.18.

P. Pokharel and P. Vaidya, “A Study of User Story in Practice,” in 2020 International Conference on Data Analytics for Business and Industry: Way Towards a Sustainable Economy, ICDABI 2020, Institute of Electrical and Electronics Engineers Inc., Oct. 2020. doi: 10.1109/ICDABI51230.2020.9325670.

B. Wake, “INVEST in Good Stories, and SMART Tasks,” XP123: Exploring Extreme Programming.

S. Jiménez, A. Alanis, C. Beltrán, R. Juárez‐Ramírez, A. Ramírez‐Noriega, and C. Tona, “USQA: A User Story Quality Analyzer prototype for supporting software engineering students,” Computer Applications in Engineering Education, vol. 31, no. 4, pp. 1014–1024, Jul. 2023, doi: 10.1002/cae.22620.

G. Lucassen, F. Dalpiaz, J. M. E. M. van der Werf, and S. Brinkkemper, “Forging high-quality User Stories: Towards a discipline for Agile Requirements,” in 2015 IEEE 23rd International Requirements Engineering Conference (RE), IEEE, Aug. 2015, pp. 126–135. doi: 10.1109/RE.2015.7320415.

G. Lucassen, F. Dalpiaz, J. M. E. M. van der Werf, and S. Brinkkemper, “Improving agile requirements: the Quality User Story framework and tool,” Requir Eng, vol. 21, no. 3, pp. 383–403, Sep. 2016, doi: 10.1007/s00766-016-0250-x.

Y. Li, J. Keung, Z. Yang, X. Ma, J. Zhang, and S. Liu, “SimAC: simulating agile collaboration to generate acceptance criteria in user story elaboration,” Automated Software Engineering, vol. 31, no. 2, p. 55, Nov. 2024, doi: 10.1007/s10515-024-00448-7.

S. S. do Nascimento, J. M. Abe, L. R. Forçan, C. C. de Oliveira, K. Nakamatsu, and A. Ari, “Improving the Process of Evaluating User Stories Using the Paraconsistent Annotated Evidential Logic Eτ,” in New Approaches for Multidimensional Signal Processing. NAMSP 2022. Smart Innovation, Systems and Technologies, vol. 332, Springer, Singapore, 2023, pp. 133–142. doi: 10.1007/978-981-19-7842-5_12.

G. Lucassen, F. Dalpiaz, J. M. E. M. van der Werf, and S. Brinkkemper, “The Use and Effectiveness of User Stories in Practice,” in REFSQ 2016: Proceedings of the 22nd International Working Conference on Requirements Engineering: Foundation for Software, 2016, pp. 205–222. doi: 10.1007/978-3-319-30282-9_14.

S. N. F. N. B. Mustaffa, J. Bin Sallim, and R. B. Mohamed, “Enhancing High-Quality User Stories with AQUSA: An Overview Study of Data Cleaning Process,” in 2021 International Conference on Software Engineering & Computer Systems and 4th International Conference on Computational Science and Information Management (ICSECS-ICOCSIM), IEEE, Aug. 2021, pp. 295–300. doi: 10.1109/ICSECS52883.2021.00060.

E. Scott, T. Tõemets, and D. Pfahl, “An Empirical Study of User Story Quality and Its Impact on Open Source Project Performance,” in Software Quality: Future Perspectives on Software Engineering Quality. SWQD 2021, vol. 404, Springer, Cham, 2021, pp. 119–138. doi: 10.1007/978-3-030-65854-0_10.

J. Bernardo, J. Frota, T. Pessoa, M. Pessoa, B. Gadelha, and T. Conte, “A Journey Through Cyberium: Creating User Stories for a Dystopic World to Teach Requirements Engineering,” in 2025 IEEE/ACM 37th International Conference on Software Engineering Education and Training (CSEE&T), IEEE, Apr. 2025, pp. 276–286. doi: 10.1109/CSEET66350.2025.00035.

A. Brockenbrough and D. Salinas, “Using Generative AI to Create User Stories in the Software Engineering Classroom,” in 2024 36th International Conference on Software Engineering Education and Training (CSEE&T), IEEE, Jul. 2024, pp. 1–5. doi: 10.1109/CSEET62301.2024.10662994.

K. Ronanki, B. Cabrero-Daniel, and C. Berger, “ChatGPT as a Tool for User Story Quality Evaluation: Trustworthy Out of the Box?,” in Agile Processes in Software Engineering and Extreme Programming - Workshops, P. Kruchten and P. Gregory, Eds., Amsterdam: Springer, Cham, 2024, ch. AI-assisted Agile, pp. 173–181. doi: 10.1007/978-3-031-48550-3_17.

O. Abed, K. Nebe, and A. B. Abdellatif, “AI-Generated User Stories Supporting Human-Centred Development: An Investigation on Quality,” in HCI International 2024 Posters. HCII 2024. Communications in Computer and Information Science, vol. 2120, Springer, Cham, 2024, pp. 3–13. doi: 10.1007/978-3-031-62110-9_1.

E. Trisnawati, I. K. Raharjana, T. Taufik, A. H. Basori, A. B. F. Mansur, and N. A. Alghanmi, “Analyzing Variances in User Story Characteristics: A Comparative Study of Stakeholders with Diverse Domain and Technical Knowledge in Software Requirements Elicitation,” Journal of Information Systems Engineering and Business Intelligence, vol. 10, no. 1, pp. 110–125, Feb. 2024, doi: 10.20473/jisebi.10.1.110-125.

A. Ananjeva, J. S. Persson, and A. Bruun, “Integrating UX work with agile development through user stories: An action research study in a small software company,” Journal of Systems and Software, vol. 170, no. December 2020, Dec. 2020, doi: 10.1016/j.jss.2020.110785.

X. Xu, Y. Dou, L. Qian, J. Jiang, K. Yang, and Y. Tan, “Quality improvement method for high-end equipment’s functional requirements based on user stories,” Advanced Engineering Informatics, vol. 56, Apr. 2023, doi: 10.1016/j.aei.2023.102017.

T. O. Hodson, “Root-mean-square error (RMSE) or mean absolute error (MAE): when to use them or not,” Geosci Model Dev, vol. 15, no. 14, pp. 5481–5487, Jul. 2022, doi: 10.5194/gmd-15-5481-2022.

E. Pinsky and S. Klawansky, “MAD (about median) vs. quantile-based alternatives for classical standard deviation, skewness, and kurtosis,” Front Appl Math Stat, vol. 9, Jun. 2023, doi: 10.3389/fams.2023.1206537.

G. D. Kader, “Means and MADs,” Mathematics Teaching in the Middle School, vol. 4, no. 6, pp. 398–403, Mar. 1999, doi: 10.5951/MTMS.4.6.0398.

E. González-Estrada and W. Cosmes, “Shapiro–Wilk test for skew normal distributions based on data transformations,” J Stat Comput Simul, vol. 89, no. 17, pp. 3258–3272, Nov. 2019, doi: 10.1080/00949655.2019.1658763.

S. S. Shapiro and M. B. Wilk, “An Analysis of Variance Test for Normality (Complete Samples),” Biometrika, vol. 52, no. 3/4, p. 591, Dec. 1965, doi: 10.2307/2333709.

R. W. Emerson, “Mann-Whitney U test and t-test,” J Vis Impair Blind, vol. 117, no. 1, pp. 99–100, Jan. 2023, doi: 10.1177/0145482X221150592.

J. C. F. de Winter and D. Dodou, “Five-Point Likert Items: t test versus Mann-Whitney-Wilcoxon,” Practical Assessment, Research & Evaluation, vol. 15, no. 12, 2010.

N. K. Malhotra, D. Nunan, and D. F. Birks, Marketing research: an applied approach, Fifth Edition. New York: Pearson, 2017.

T. D. Gauthier and M. E. Hawley, “Statistical Method,” in Introduction to Environmental Forensics, Elsevier, 2007, pp. 129–183. doi: 10.1016/B978-012369522-2/50006-3.

Downloads

Published

2025-10-25

Issue

Section

Computer Science Applications

How to Cite

Zul, M. I., Yasin, S. M., & Sahid, D. S. S. (2025). Student-Generated User Story Quality: A Study on Practitioner and ChatGPT Evaluation. Jurnal RESTI (Rekayasa Sistem Dan Teknologi Informasi), 9(5), 1163-1172. https://doi.org/10.29207/resti.v9i5.6950