Ingredients Identification Through Label Scanning Using PaddleOCR and ChatGPT for Information Retrieval

Ahmad Wahyu Rosyadi; Siti Ma’shumah; Muhammad Qomaruz Zaman; Moh. Rizki Fajar

doi:10.29207/resti.v8i6.6119

Ahmad Wahyu Rosyadi Universitas Qomaruddin
Siti Ma’shumah Universitas Qomaruddin
Muhammad Qomaruz Zaman Universitas Qomaruddin
Moh. Rizki Fajar Universitas Qomaruddin

DOI: https://doi.org/10.29207/resti.v8i6.6119

Keywords: ChatGPT, food identification, information retrieval, optical character recognition, PaddleOCR

Abstract

Human health depends on choosing food ingredients that align with dietary needs and avoid allergens. However, consumers often encounter unfamiliar ingredients that require additional information. Traditionally, they search online by typing in the ingredient's name which can be time-consuming and may not yield relevant results. Therefore, a system to identify and display ingredient information is necessary. This study proposes a new system that identifies ingredients by scanning the composition label on packaging using PaddleOCR and retrieving information through ChatGPT on a smartphone. The process begins with capturing an image of the composition label. Then PaddleOCR is employed to extract text from the scanned label, enabling identification of the listed ingredients. Subsequently, ChatGPT retrieves detailed information about the desired ingredients and displays it, allowing users to easily understand the ingredients. The system's effectiveness in text recognition is assessed using the character error rate (CER). The results show robust performance by achieving an average CER of 0.14, with flat packaging reaching an impressive CER of 0.05. Additionally, the system's usability was assessed through pilot testing which received significant positive user feedback achieving 4.37 satisfaction level on Likert scale, particularly regarding the clarity and relevance of the ingredient information provided.

Downloads

Download data is not yet available.

References

B. Ozhan and B. Tugrul, “Analysis of Turkish cuisine flavors network,” Int. J. Food Sci. Technol., pp. 908–915, 2023, doi: 10.1111/ijfs.16849.

M. Maringer, N. Wisse-Voorwinden, P. V. t. Veer, and A. Geelen, “Food identification by barcode scanning in the Netherlands: A quality assessment of labelled food product databases underlying popular nutrition applications,” Public Health Nutr., vol. 22, no. 7, pp. 1215–1222, 2019, doi: 10.1017/S136898001800157X.

A. G. Arens-Volland, L. Spassova, and T. Bohn, “Promising approaches of computer-supported dietary assessment and management-Current research status and available applications.,” Int. J. Med. Inform., vol. 84, no. 12, pp. 997–1008, Dec. 2015, doi: 10.1016/j.ijmedinf.2015.08.006.

M. Rusin, E. Arsand, and G. Hartvigsen, “Functionalities and input methods for recording food intake: a systematic review.,” Int. J. Med. Inform., vol. 82, no. 8, pp. 653–664, Aug. 2013, doi: 10.1016/j.ijmedinf.2013.01.007.

R. Steele, “An overview of the state of the art of automated capture of dietary intake information.,” Crit. Rev. Food Sci. Nutr., vol. 55, no. 13, pp. 1929–1938, 2015, doi: 10.1080/10408398.2013.765828.

C. Byrd-Bredbenner and C. A. Bredbenner, “Assessing the home food environment nutrient supply using mobile barcode (Universal Product Code) scanning technology,” Nutr. Food Sci., vol. 40, no. 3, pp. 305–313, 2010, doi: 10.1108/00346651011044005.

P. H. Gosavi, P. Kawale, P. Kadam, and A. Manjhi, “Nutrilens : Barcode Scanner,” vol. 13, no. 4, pp. 90–97, 2024.

C. Li et al., “Dive into OCR,” 2022.

K. V Ujwal Karanth, A. T. Sujan, Y. R. Thanay Kumar, S. Joshi, K. P. Asha Rani, and S. Gowrishankar, “Breaking Barriers in Text Analysis: Leveraging Lightweight OCR and Innovative Technologies for Efficient Text Analysis,” pp. 359–366, 2024, doi: 10.1109/icacrs58579.2023.10404305.

H. Tao and C. Mingyang, “Character Recognition and Printing Quality Inspection of Chips under Packaging Film,” Proc. - 2022 Int. Conf. Comput. Eng. Artif. Intell. ICCEAI 2022, pp. 139–143, 2022, doi: 10.1109/ICCEAI55464.2022.00037.

Z. Hu, X. He, Y. Lin, Y. Zhao, Z. Liang, and J. Wu, “Intelligent Point Reader Based on Machine Vision and Speech Processing,” 2023 IEEE Int. Conf. Electr. Autom. Comput. Eng. ICEACE 2023, pp. 975–983, 2023, doi: 10.1109/ICEACE60673.2023.10442012.

J. Krishna Manipatruni et al., “Leveraging Artificial Intelligence for Simplified Invoice Automation: Paddle OCR-based Text Extraction from Invoices,” Int. J. Innov. Sci. Res. Technol., vol. 8, no. 9, 2023, [Online]. Available: www.ijisrt.com

D. Zhang and Y. Li, “Research and Application of Health Code Recognition Based on Paddle OCR under the Background of Epidemic Prevention and Control,” J. Artif. Intell. Pract., vol. 6, no. 1, pp. 9–16, 2023, doi: 10.23977/jaip.2023.060102.

K. Li et al., “Eval-GCSC: A New Metric for Evaluating ChatGPT’s Performance in Chinese Spelling Correction,” 2023, [Online]. Available: http://arxiv.org/abs/2311.08219

A. Tlili et al., “What if the devil is my guardian angel: ChatGPT as a case study of using chatbots in education,” Smart Learn. Environ., vol. 10, no. 1, 2023, doi: 10.1186/s40561-023-00237-x.

M. Koo, “The Importance of Proper Use of ChatGPT in Medical Writing,” Radiology, vol. 307, no. 3, p. 7945, 2023, doi: 10.1148/RADIOL.230312.

Y. Huang and J. X. Huang, “Exploring ChatGPT for next-generation information retrieval: Opportunities and challenges,” Web Intell., vol. 22, no. 1, pp. 31–44, 2024, doi: 10.3233/WEB-230363.

M. Pan, J. Wang, J. X. Huang, A. J. Huang, Q. Chen, and J. Chen, “A probabilistic framework for integrating sentence-level semantics via BERT into pseudo-relevance feedback,” Inf. Process. Manag., vol. 59, no. 1, p. 102734, 2022, doi: https://doi.org/10.1016/j.ipm.2021.102734.

A. W. Rosyadi and N. Suciati, “Median filter for transition region refinement in image segmentation,” JUTI J. Ilm. Teknol. Inf., vol. 16, no. 2, pp. 158–170, 2018.

A. W. Rosyadi and N. Suciati, “Image Segmentation Using Transition Region and K-Means Clustering,” IAENG Int. J. Comput. Sci., vol. 47, no. 1, 2020.

H. Zhang et al., “Identification of Illegal Outdoor Advertisements Based on CLIP Fine-Tuning and OCR Technology,” IEEE Access, vol. 12, no. July, pp. 92976–92987, 2024, doi: 10.1109/ACCESS.2024.3424258.

F. Year, P. Final, P. T. Hieu, and V. Van Nghiep, “A Deep Learning Model for Helmet Detection and Automatic License Plate Recognition,” no. December, 2023.

Y. Du et al., “SVTR: Scene Text Recognition with a Single Visual Model,” IJCAI Int. Jt. Conf. Artif. Intell., vol. 1, no. c, pp. 884–890, 2022, doi: 10.24963/ijcai.2022/124.

C. Cui et al., “PP-LCNet: A Lightweight CPU Convolutional Neural Network,” 2021, [Online]. Available: http://arxiv.org/abs/2109.15099

A. Z. Arifin, M. Z. Abdullah, A. W. Rosyadi, D. I. Ulumi, A. Wahib, and R. W. Sholikah, “Sentence extraction based on sentence distribution and part of speech tagging for multi-document summarization,” Telkomnika (Telecommunication Comput. Electron. Control., vol. 16, no. 2, pp. 843–851, 2018, doi: 10.12928/TELKOMNIKA.v16i2.8431.

A. Rao et al., “Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow: Development and Usability Study,” J. Med. Internet Res., vol. 25, pp. 1–9, 2023, doi: 10.2196/48659.

S. S. Biswas, “Role of Chat GPT in Public Health,” Ann. Biomed. Eng., vol. 51, no. 5, pp. 868–869, 2023, doi: 10.1007/s10439-023-03172-7.

R. Xu, Y. (Katherine) Feng, and H. Chen, “ChatGPT vs. Google: A Comparative Study of Search Performance and User Experience,” SSRN Electron. J., no. July, 2023, doi: 10.2139/ssrn.4498671.

G. Leifert, R. Labahn, T. Gruning, and S. Leifert, “End-to-end measure for text recognition,” Proc. Int. Conf. Doc. Anal. Recognition, ICDAR, no. Ld, pp. 1424–1431, 2019, doi: 10.1109/ICDAR.2019.00-16.

W. Suryn, Software Quality Engineering: A Practitioner’s Approach, vol. 9781118592. 2014. doi: 10.1002/9781118830208.

C. M. Barnum, Usability Testing Essentials: Ready, Set…Test! 2020. doi: 10.1016/B978-0-12-816942-1.01001-7.

Y. Sari, M. Arafah, and Novitasari, “Evaluasi Usability Sistem Informasi Akademik Dosen Menggunakan User Experience Questionnaire dan Heuristic Walkthrough,” J. RESTI (Rekayasa Sist. dan Teknol. Informasi), vol. 5, no. 2, pp. 247–253, 2021, doi: 10.29207/resti.v5i2.3022.

Darussalam and G. Arief, “Jurnal Resti,” Resti, vol. 1, no. 1, pp. 19–25, 2018.