Leveraging Assistive Image Captioning on VizWiz through Multi-Caption Supervision

Fauzan Firdaus; Ema Rachmawati; Angel Metanosa Afinda; Edward Ferdian; Muhammad Maulana Yusuf

doi:10.29207/resti.v10i5.7472

Authors

Fauzan Firdaus Telkom University
Ema Rachmawati Telkom University
Angel Metanosa Afinda Telkom University
Edward Ferdian Telkom University
Muhammad Maulana Yusuf Telkom University

DOI:

https://doi.org/10.29207/resti.v10i5.7472

Keywords:

image captioning, multi-caption training, VizWiz dataset

Abstract

Most image captioning datasets provide several human captions for the same image, yet model training is frequently carried out in a way that effectively treats each image as having only one caption. Our study revisits this mismatch by examining the impact of training with multiple captions per image on caption generation performance. For this purpose, we utilized the VizWiz-Captions dataset, which reflects a real assistive scenario with images taken by blind or visually impaired users with five reference captions per image. We evaluated two conventional CNN-based encoder–decoder models, namely CNN–LSTM and CNN–GRU. In both models, the visual encoder was fixed using InceptionV3 image features, while the recurrent decoder was varied between LSTM and GRU. We performed experiments by using training data by varying the number of captions used per image from one to five (k = 1–5), while keeping the overall pipeline unchanged. The models were evaluated using standard multi-reference metrics (BLEU-1 to BLEU-4, CIDEr, and SPICE). Across both baselines, performance generally improves when more captions are included in training, although the gains are not perfectly monotonic for every metric. The strongest results are observed at k = 5. For CNN–LSTM, the best setting achieved BLEU-4 = 0.175, CIDEr = 0.206, and SPICE = 0.120. On the other hand, CNN–GRU achieved BLEU-4 = 0.192, CIDEr = 0.226, and SPICE = 0.119. These findings suggest that, even without architectural changes, simply leveraging the available caption diversity during training can provide a practical boost for VizWiz-style captioning tasks.

Downloads

Download data is not yet available.

References

[1] D. Gurari, Y. Zhao, M. Zhang, and N. Bhattacharya, “Captioning Images Taken by People Who Are Blind,” Jul. 15, 2020, arXiv: arXiv:2002.08565. doi: https://doi.org/10.48550/arXiv.2002.08565.

[2] P. Dognin et al., “Image Captioning as an Assistive Technology: Lessons Learned from VizWiz 2020 Challenge,” J Artif Int Res, vol. 73, May 2022, doi: https://doi.org/10.1613/jair.1.13113.

[3] H. Sharma, “A Survey on Image Captioning datasets and Evaluation Metrics,” IOP Conf. Ser. Mater. Sci. Eng., vol. 1116, no. 1, p. 012184, Apr. 2021, doi: https://doi.org/10.1088/1757-899X/1116/1/012184.

[4] Y. Mao et al., “Improving Reference-Based Distinctive Image Captioning with Contrastive Rewards,” ACM Trans Multimed. Comput Commun Appl, vol. 20, no. 12, p. 390:1-390:24, Nov. 2024, doi: https://doi.org/10.1145/3694683.

[5] A. Radford et al., “Learning Transferable Visual Models From Natural Language Supervision,” Feb. 26, 2021, arXiv: arXiv:2103.00020. doi: https://doi.org/10.48550/arXiv.2103.00020.

[6] Z. Sun et al., “Alpha-CLIP: A CLIP Model Focusing on Wherever You Want,” Dec. 13, 2023, arXiv: arXiv:2312.03818. doi: https://doi.org/10.48550/arXiv.2312.03818.

[7] J. Li, D. Li, S. Savarese, and S. Hoi, “BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models,” Jun. 15, 2023, arXiv: arXiv:2301.12597. doi: https://doi.org/10.48550/arXiv.2301.12597.

[8] J.-B. Alayrac et al., “Flamingo: a Visual Language Model for Few-Shot Learning” Nov. 15, 2022, arXiv: arXiv:2204.14198. doi: https://doi.org/10.48550/arXiv.2204.14198.

[9] W. Dai et al., “InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning,” Jun. 15, 2023, arXiv: arXiv:2305.06500. doi: https://doi.org/10.48550/arXiv.2305.06500.

[10] H. Liu, C. Li, Y. Li, and Y. J. Lee, “Improved Baselines with Visual Instruction Tuning,” May 15, 2024, arXiv: arXiv:2310.03744. doi: https://doi.org/10.48550/arXiv.2310.03744.

[11] Q. Ye, X. Zeng, F. Li, C. Li, and H. Fan, “Painting with Words: Elevating Detailed Image Captioning with Benchmark and Alignment Learning,” presented at the The Thirteenth International Conference on Learning Representations, Oct. 2024. Accessed: Jan. 05, 2026. [Online]. Available: https://openreview.net/forum?id=636M0nNbPs

[12] L. Yu, M. Nikandrou, J. Jin, and V. Rieser, “Quality-agnostic Image Captioning to Safely Assist People with Vision Impairment,” in Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, Macau, SAR China: International Joint Conferences on Artificial Intelligence Organization, Aug. 2023, pp. 6281–6289. doi: https://doi.org/10.24963/ijcai.2023/697.

[13] A. Thobhani et al., “A Survey on Enhancing Image Captioning with Advanced Strategies and Techniques,” Comput. Model. Eng. Sci., vol. 142, no. 3, pp. 2247–2280, Mar. 2025, doi: https://doi.org/10.32604/cmes.2025.059192.

[14] Kavitha Soppari, Pakide Kavya, Kotla Pranay Teja, and Bethi Pavan Sai, “A survey on image captioning methods,” World J. Adv. Res. Rev., vol. 26, no. 2, pp. 3134–3143, May 2025, doi: https://doi.org/10.30574/wjarr.2025.26.2.1705.

[15] A. Abdussalam, Z. Ye, A. Hawbani, M. Al-Qatf, and R. Khan, “NumCap: A Number-controlled Multi-caption Image Captioning Network,” ACM Trans Multimed. Comput Commun Appl, vol. 19, no. 4, p. 148:1-148:24, Feb. 2023, doi: https://doi.org/10.1145/3576927.

[16] M. U. Hadi et al., “A Survey on Large Language Models: Applications, Challenges, Limitations, and Practical Usage”, Accessed: Jan. 14, 2026, doi: https://doi.org/10.36227/techrxiv.23589741.v1.

[17] A. Joshi, K. Kalal, D. Bhandare, V. Patil, U. Kulkarni, and S. M. Meena, “Image Captioning Using CNN-LSTM,” in Advances in Communication and Applications, N. R. Shetty, N. H. Prasad, and H. C. Nagaraj, Eds., Singapore: Springer Nature, 2024, pp. 421–433. doi: https://doi.org/10.1007/978-981-99-7633-1_32.

[18] A. A. E. Osman, M. A. W. Shalaby, M. M. Soliman, and K. M. Elsayed, “A Survey on Attention-Based Models for Image Captioning,” Int. J. Adv. Comput. Sci. Appl. IJACSA, vol. 14, no. 2, Feb. 2023, doi: https://doi.org/10.14569/IJACSA.2023.0140249.

[19] L. Celona, S. Bianco, M. Donzella, and P. Napoletano, “Improving image captioning descriptiveness by ranking and LLM-based fusion,” Neural Comput. Appl., vol. 37, no. 32, pp. 27279–27299, Nov. 2025, doi: https://doi.org/10.1007/s00521-025-11672-x.

[20] V. Zouhar and O. Bojar, “Quality and Quantity of Machine Translation References for Automatic Metrics,” Apr. 10, 2024, arXiv: arXiv:2401.01283. doi: https://doi.org/10.48550/arXiv.2401.01283.

[21] S. Casola, Y. J. Liu, S. Peng, O. Kraus, A. Gatt, and B. Plank, “References Matter: Investigating the Impact of Reference Set Variation on Summarization Evaluation,” in Proceedings of the 18th International Natural Language Generation Conference, L. Flek, S. Narayan, L. H. Phương, and J. Pei, Eds., Hanoi, Vietnam: Association for Computational Linguistics, Oct. 2025, pp. 274–291. Accessed: Jan. 14, 2026. [Online]. Available: https://aclanthology.org/2025.inlg-main.18/

[22] T. Tang et al., “Not All Metrics Are Guilty: Improving NLG Evaluation by Diversifying References,” in Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), K. Duh, H. Gomez, and S. Bethard, Eds., Mexico City, Mexico: Association for Computational Linguistics, Jun. 2024, pp. 6596–6610. doi: https://doi.org/10.18653/v1/2024.naacl-long.367.

[23] A. Thobhani et al., “A novel image captioning model with visual-semantic similarities and visual representations re-weighting,” J. King Saud Univ. - Comput. Inf. Sci., vol. 36, no. 7, p. 102127, Sep. 2024, doi: https://doi.org/10.1016/j.jksuci.2024.102127.

[24] Y. H. Zaidan and J. Waleed, “Image Captioning Generation Using Inception V3 and Attention Mechanism,” J. Al-Qadisiyah Comput. Sci. Math., vol. 15, no. 2, p. Page 1-10, Sep. 2023, doi: https://doi.org/10.29304/jqcm.2023.15.2.1228.

[25] A. K. Poddar and Dr. R. Rani, “Hybrid Architecture using CNN and LSTM for Image Captioning in Hindi Language,” Procedia Comput. Sci., vol. 218, pp. 686–696, Jan. 2023, doi: https://doi.org/10.1016/j.procs.2023.01.049.

[26] Y. Geng, H. Mei, X. Xue, and X. Zhang, “Image-Caption Model Based on Fusion Feature,” Appl. Sci., vol. 12, no. 19, p. 9861, Jan. 2022, doi: https://doi.org/10.3390/app12199861.

[27] N. M. Khassaf and N. H. M. Ali, “Improving Pre-trained CNN-LSTM Models for Image Captioning with Hyper-Parameter Optimization,” Eng. Technol. Appl. Sci. Res., vol. 14, no. 5, pp. 17337–17343, Oct. 2024, doi: https://doi.org/10.48084/etasr.8455.

[28] K. R. Suresh, A. Jarapala, and P. V. Sudeep, “Image Captioning Encoder–Decoder Models Using CNN-RNN Architectures: A Comparative Study,” Circuits Syst. Signal Process., vol. 41, no. 10, pp. 5719–5742, Oct. 2022, doi: https://doi.org/10.1007/s00034-022-02050-2.

[29] R. Khan, M. S. Islam, K. Kanwal, M. Iqbal, M. I. Hossain, and Z. Ye, “A Deep Neural Framework for Image Caption Generation Using GRU-Based Attention Mechanism,” Mar. 03, 2022, arXiv: arXiv:2203.01594. doi: https://doi.org/10.48550/arXiv.2203.01594.

[30] U. Berger, G. Stanovsky, O. Abend, and L. Frermann, “Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis,” Sep. 14, 2025, arXiv: arXiv:2408.04909. doi: https://doi.org/10.48550/arXiv.2408.04909.

[31] O. González-Chávez, G. Ruiz, D. Moctezuma, and T. Ramirez-delReal, “Are metrics measuring what they should? An evaluation of Image Captioning task metrics,” Signal Process. Image Commun., vol. 120, p. 117071, Jan. 2024, doi: https://doi.org/10.1016/j.image.2023.117071.

[32] A. de S. Inácio and H. S. Lopes, “Evaluation metrics for video captioning: A survey,” Mach. Learn. Appl., vol. 13, p. 100488, Sep. 2023, doi: https://doi.org/10.1016/j.mlwa.2023.100488.