An Empirical Evaluation of ChatGPT as an Automated Machine Learning Code Generator for Image Classification
DOI:
https://doi.org/10.29207/joseit.v4i2.7174Keywords:
ChatGPT, machine learning, code generation, image classification, artificial intelligenceAbstract
The emergence of large language models such as ChatGPT has created unprecedented opportunities for automating software development processes, particularly within the machine learning domain. This study aims to empirically evaluate the effectiveness of ChatGPT in generating machine learning code for image classification tasks using the Keras framework. The research employs an experimental methodology utilizing the MNIST dataset, comprising 70,000 handwritten digit images. A systematic series of experiments was conducted through progressive prompting strategies, ranging from basic model construction to comprehensive evaluation protocols. The findings demonstrate that ChatGPT successfully generated 100% executable code without errors, with the resulting models achieving 99% accuracy on the test dataset. A notable discovery emerged in the form of "intelligent deviation" phenomena, wherein ChatGPT autonomously provided Convolutional Neural Network (CNN) architectures despite explicit requests for fully connected layers, demonstrating sophisticated contextual understanding. The generated code quality met professional standards with robust multi-library integration capabilities. This research provides the first systematic empirical contribution regarding large language model capabilities in machine learning code generation, offering significant implications for democratizing artificial intelligence technology access within educational and research contexts.
References
T. Teubner, C. M. Flath, C. Weinhardt, W. van der Aalst, and O. Hinz, “Welcome to the Era of ChatGPT et al.: The Prospects of Large Language Models,” 2023. doi: 10.1007/s12599-023-00795-x.
E. Kasneci et al., “ChatGPT for good? On opportunities and challenges of large language models for education,” Learn Individ Differ, vol. 103, 2023, doi: 10.1016/j.lindif.2023.102274.
T. B. Brown et al., “Language models are few-shot learners,” in Advances in Neural Information Processing Systems, 2020.
J. Gallifant et al., “Peer review of GPT-4 technical report and systems card,” 2024. doi: 10.1371/journal.pdig.0000417.
Y. Kim, “Convolutional neural networks for sentence classification,” in EMNLP 2014 - 2014 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference, 2014. doi: 10.3115/v1/d14-1181.
Y. Chen, Z. Lin, X. Zhao, G. Wang, and Y. Gu, “Deep learning-based classification of hyperspectral data,” IEEE J Sel Top Appl Earth Obs Remote Sens, vol. 7, no. 6, 2014, doi: 10.1109/JSTARS.2014.2329330.
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,” Commun ACM, vol. 60, no. 6, 2017, doi: 10.1145/3065386.
D. Learning, “Deep Learning - Goodfellow,” Nature, vol. 26, no. 7553, 2016.
X. He, K. Zhao, and X. Chu, “AutoML: A survey of the state-of-the-art,” Knowl Based Syst, vol. 212, 2021, doi: 10.1016/j.knosys.2020.106622.
M. Feurer, A. Klein, K. Eggensperger, J. T. Springenberg, M. Blum, and F. Hutter, “Efficient and robust automated machine learning,” in Advances in Neural Information Processing Systems, 2015.
R. Watrianthos, S. Triono Ahmad, and M. Muskhir, “Charting the Growth and Structure of Early ChatGPT-Education Research: A Bibliometric Study,” Journal of Information Technology Education: Innovations in Practice, vol. 22, pp. 235–253, 2023, doi: 10.28945/5221.
“ChatGPT: Educational Artificial Intelligence,” International Journal of Advanced Trends in Computer Science and Engineering, vol. 12, no. 2, pp. 84–91, Apr. 2023, doi: 10.30534/ijatcse/2023/091222023.
L. Huang, J. Zhao, B. Zhu, H. Chen, and S. Vanden Broucke, “An Experimental Investigation of Calibration Techniques for Imbalanced Data,” IEEE Access, vol. 8, 2020, doi: 10.1109/ACCESS.2020.3008150.
M. Zhong, Z. Wang, G. Liu, Y. Chen, H. Liu, and R. Wu, “CodeGen-Test: An Automatic Code Generation Model Integrating Program Test Information,” in 2023 2nd International Conference on Cloud Computing, Big Data Application and Software Engineering, CBASE 2023, 2023. doi: 10.1109/CBASE60015.2023.10439105.
O. Russakovsky et al., “ImageNet Large Scale Visual Recognition Challenge,” Int J Comput Vis, vol. 115, no. 3, 2015, doi: 10.1007/s11263-015-0816-y.
L. Deng, “The MNIST database of handwritten digit images for machine learning research,” IEEE Signal Process Mag, vol. 29, no. 6, 2012, doi: 10.1109/MSP.2012.2211477.
J. D. Weisz et al., “Perfection Not Required? Human-AI Partnerships in Code Translation,” in International Conference on Intelligent User Interfaces, Proceedings IUI, 2021. doi: 10.1145/3397481.3450656.
N. Yanes, A. M. Mostafa, M. Ezz, and S. N. Almuayqil, “A machine learning-based recommender system for improving students learning experiences,” IEEE Access, vol. 8, 2020, doi: 10.1109/ACCESS.2020.3036336.
S. Ding, H. Zhao, Y. Zhang, X. Xu, and R. Nie, “Extreme learning machine: algorithm, theory and applications,” Artif Intell Rev, vol. 44, no. 1, pp. 103–115, 2015, doi: 10.1007/s10462-013-9405-z.
R. Kumari and S. Kr., “Machine Learning: A Review on Binary Classification,” Int J Comput Appl, vol. 160, no. 7, 2017, doi: 10.5120/ijca2017913083.
N. Ketkar and J. Moolayil, Deep Learning with Python. 2021. doi: 10.1007/978-1-4842-5364-9.
J. Hung et al., “Keras R-CNN: Library for cell detection in biological images using deep neural networks,” BMC Bioinformatics, vol. 21, no. 1, 2020, doi: 10.1186/s12859-020-03635-x.
Downloads
Published
Issue
Section
License
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under Creative Commons Attribution 4.0 International License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (Refer to The Effect of Open Access).
