Towards Generating Unit Test Codes Using Generative Adversarial Networks
Abstract
Unit testing is one of the critical software development steps to ensure the software’s quality. Unit testing is often neglected despite its importance since it requires a significant amount of time and effort from the software developers to write them. Existing automated testing generating systems from past research still have shortcomings due to the limitations of the Genetic Algorithm (GA) in generating the appropriate unit test codes. This study explores the feasibility of using Generative Adversarial Networks (GAN) models to generate unit test code with the ability of GAN to cover GA’s drawbacks. We perform experimentations using four state-of-the-art GAN models to generate basic unit test codes and compare the results by analyzing the generated output codes using novel metrics proposed from past studies and performing a qualitative evaluation of the generated outputs. The results show that the generated codes have satisfactory quality scores (BLEU-2 of around 99%) from the models and adequate diversity scores (NLL-Div and NLL-Gen) in most models. Our study shows positive indications and potential in the use of GAN for automatic unit test code generation and suggests recommendations for future studies in GAN-based unit test code generation systems
Downloads
References
P. Runeson, "A survey of unit testing practices," in IEEE Software, vol. 23, no. 4, pp. 22-29, July-Aug. 2006.
https://doi.org/10.1109/MS.2006.91
M. F. Aniche, T. M. Ferreira, en M. A. Gerosa, “What concerns beginner test-driven development practitioners: a qualitative analysis of opinions in an agile conference”, in 2nd Brazilian Workshop on Agile Methods, 2011, vol 19, bl 22.
https://www.ime.usp.br/~aniche/files/wbma2011.pdf
S. Romano, D. Fucci, M. T. Baldassarre, D. Caivano, and G. Scanniello, “An empirical assessment on affective reactions of novice developers when applying test-driven development,” Product-Focused Software Process Improvement, pp. 3–19, 2019.
https://doi.org/10.1007/978-3-030-35333-9_1
G. Fraser and A. Arcuri, "Whole Test Suite Generation," in IEEE Transactions on Software Engineering, vol. 39, no. 2, pp. 276-291, Feb. 2013.
https://doi.org/10.1109/TSE.2012.14
A. Panichella, F. M. Kifetew and P. Tonella, "Automated Test Case Generation as a Many-Objective Optimisation Problem with Dynamic Selection of the Targets," in IEEE Transactions on Software Engineering, vol. 44, no. 2, pp. 122-158, 1 Feb. 2018.
https://doi.org/10.1109/TSE.2017.2663435
S. Lukasczyk, F. Kroiß, en G. Fraser, “Automated Unit Test Generation for Python”, in Search-Based Software Engineering, 2020, bll 9–24.
https://doi.org/10.1007/978-3-030-59762-7_2
S. Katoch, S. S. Chauhan, en V. Kumar, “A review on genetic algorithm: past, present, and future”, Multimedia Tools and Applications, vol 80, no 5, bll 8091–8126, Feb 2021.
https://doi.org/10.1007/s11042-020-10139-6
M. M. Almasi, H. Hemmati, G. Fraser, A. Arcuri and J. Benefelds, "An Industrial Evaluation of Unit Test Generation: Finding Real Faults in a Financial Application," 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP), 2017, pp. 263-272.
https://doi.org/10.1109/ICSE-SEIP.2017.27
I. J. Goodfellow et al., “Generative Adversarial Nets”, in Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2, Montreal, Canada, 2014, bll 2672–2680.
http://dx.doi.org/10.1145/3422622
Y. Li et al., "StoryGAN: A Sequential Conditional GAN for Story Visualization," 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 6322-6331.
https://doi.org/10.1109/CVPR.2019.00649
Y. Balaji, M. R. Min, B. Bai, R. Chellappa, en H. P. Graf, “Conditional GAN with Discriminative Filter Generation for Text-to-Video Synthesis”, in Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, 7 2019, bll 1995–2001.
https://doi.org/10.24963/ijcai.2019/276
Y. Li, M. Min, D. Shen, D. Carlson, en L. Carin, “Video generation from text”, in Proceedings of the AAAI Conference on Artificial Intelligence, 2018, vol 32.
https://ojs.aaai.org/index.php/AAAI/article/view/12233
G. H. de Rosa en J. P. Papa, “A survey on text generation using generative adversarial networks”, Pattern Recognition, vol 119, bl 108098, 2021.
https://doi.org/10.1016/j.patcog.2021.108098
A. Creswell, T. White, V. Dumoulin, K. Arulkumaran, B. Sengupta and A. A. Bharath, "Generative Adversarial Networks: An Overview," in IEEE Signal Processing Magazine, vol. 35, no. 1, pp. 53-65, Jan. 2018.
https://doi.org/10.1109/MSP.2017.2765202
B. Dai, S. Fidler, R. Urtasun and D. Lin, "Towards Diverse and Natural Image Descriptions via a Conditional GAN," 2017 IEEE International Conference on Computer Vision (ICCV), 2017, pp. 2989-2998.
https://doi.org/10.1109/ICCV.2017.323
J. Xu, X. Ren, J. Lin, en X. Sun, “Diversity-Promoting GAN: A Cross-Entropy Based Generative Adversarial Network for Diversified Text Generation”, in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, bll 3940–3949.
http://dx.doi.org/10.18653/v1/D18-1428
W. Nie, N. Narodytska, en A. Patel, “RelGAN: Relational Generative Adversarial Networks for Text Generation”, in International Conference on Learning Representations, 2019.
https://openreview.net/forum?id=rJedV3R5tm
X. Liu, X. Kong, L. Liu and K. Chiang, "TreeGAN: Syntax-Aware Sequence Generation with Generative Adversarial Networks," 2018 IEEE International Conference on Data Mining (ICDM), 2018, pp. 1140-1145.
https://doi.org/10.1109/ICDM.2018.00149
L. Yu, W. Zhang, J. Wang, en Y. Yu, “SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient”, in Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, 2017, bll 2852–2858.
https://doi.org/10.48550/arXiv.1609.05473
T. Che et al., “Maximum-likelihood augmented discrete generative adversarial networks”, arXiv preprint arXiv:1702. 07983, 2017.
https://doi.org/10.48550/arXiv.1702.07983
R. D. Hjelm, A. P. Jacob, A. Trischler, G. Che, K. Cho, en Y. Bengio, “Boundary Seeking GANs”, in International Conference on Learning Representations, 2018.
https://openreview.net/forum?id=rkTS8lZAb
X. Zhang, J. Zhao, en Y. LeCun, “Character-level Convolutional Networks for Text Classification”, in Advances in Neural Information Processing Systems, 2015, vol 28.
https://proceedings.neurips.cc/paper/2015/hash/250cf8b51c773f3f8dc8b4be867a9a02-Abstract.html
Z. Li, T. Xia, X. Lou, K. Xu, S. Wang, en J. Xiao, “Adversarial Discrete Sequence Generation without Explicit NeuralNetworks as Discriminators”, in Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, 16--18 Apr 2019, vol 89, bll 3089–3098.
https://proceedings.mlr.press/v89/li19g.html
K. O. Burtch, “The Most Common Variable Names,” The Lone Coder - Pegasoft Canada, 18-Jul-2014. [Online]. Available: https://www.pegasoft.ca/coder/coder_july_2014.html.
P. Kawthekar, R. Rewari, en S. Bhooshan, “Evaluating generative models for text generation”. Stanford University, 2017.
https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1174/reports/2737434.pdf
K. Papineni, S. Roukos, T. Ward, en W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation”, in Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 2002, bll 311–318.
http://dx.doi.org/10.3115/1073083.1073135
Z. Liu, J. Wang, en Z. Liang, “CatGAN: Category-Aware Generative Adversarial Networks with Hierarchical Evolutionary Learning for Category Text Generation”, Proceedings of the AAAI Conference on Artificial Intelligence, vol 34, bll 8425–8432, 04 2020.
http://dx.doi.org/10.1609/aaai.v34i05.6361
H. Yao, D.-L. Zhu, B. Jiang, en P. Yu, “Negative log likelihood ratio loss for deep neural network classification”, in Proceedings of the Future Technologies Conference, 2019, bll 276–282.
https://doi.org/10.1007/978-3-030-32520-6_22
Y. Zhu et al., “Texygen: A Benchmarking Platform for Text Generation Models”, in the 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, Ann Arbor, MI, USA, 2018, bll 1097–1100.
Copyright (c) 2022 Jurnal RESTI (Rekayasa Sistem dan Teknologi Informasi)
This work is licensed under a Creative Commons Attribution 4.0 International License.
Copyright in each article belongs to the author
- The author acknowledges that the RESTI Journal (System Engineering and Information Technology) is the first publisher to publish with a license Creative Commons Attribution 4.0 International License.
- Authors can enter writing separately, arrange the non-exclusive distribution of manuscripts that have been published in this journal into other versions (eg sent to the author's institutional repository, publication in a book, etc.), by acknowledging that the manuscript has been published for the first time in the RESTI (Rekayasa Sistem dan Teknologi Informasi) journal ;
