DETR-CLIP-BLIP Integrated Framework for Context-Guided Multimodal Knowledge Graph Retrieval

Raihan Atsal  Hafizh; Kemas Rahmat Saleh  Wiharja

doi:10.29207/resti.v10i5.7647

Authors

Raihan Atsal Hafizh Telkom University
Kemas Rahmat Saleh Wiharja Telkom University

DOI:

https://doi.org/10.29207/resti.v10i5.7647

Keywords:

attribute classification, image retrieval, multimodal knowledge graph, object detection, semantic relationships, vision-language models

Abstract

Image search systems that rely on metadata tags or basic captions often miss the semantic relationships and visual details needed for complex queries. This study presents a multimodal knowledge graph framework that combines DETR for object detection, CLIP for attribute classification across 188 dimensions, and BLIP for caption generation. On the COCO 2017 validation set (5,000 images), the pipeline processed 4,906 images successfully, of which 4,243 contained the multiple objects required for relationship extraction. The resulting knowledge graph reached 99.98% relational coverage among multi-object images and 2.94 relationships per image, with 117 of the 188 attributes appearing in the data. Retrieval was evaluated under a self-consistency protocol relevance defined by the system's own top-ranked matches yielding an MRR of 0.96 for keyword-based queries and 0.29 for relationship-based queries, with Recall@50 of 88% for the latter. The framework runs on pretrained models without task-specific training and produces both structured triples and human-readable descriptions, supporting applications such as visual question answering, scene understanding, and context-aware image retrieval.

Downloads

Download data is not yet available.

References

[1] H. Hu, "COCOKG: A Multimodal Knowledge Graph Building on MSCOCO," in Advances in Natural Computation, Fuzzy Systems and Knowledge Discovery (ICNC-FSKD 2021), Lecture Notes on Data Engineering and Communications Technologies, vol. 89, Q. Xie, L. Zhao, K. Li, A. Yadav, and L. Wang, Eds. Cham: Springer, 2022, pp. 822–829. doi: 10.1007/978-3-030-89698-0_84

[2] R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Shamma, M. S. Bernstein, and L. Fei-Fei, "Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations," Int. J. Comput. Vis., vol. 123, no. 1, pp. 32-73, May 2017. doi: 10.1007/s11263-016-0981-7

[3] L. Hu, Z. Liu, Z. Zhao, L. Hou, L. Nie, and J. Li, "A Survey of Knowledge Enhanced Pre-Trained Language Models," IEEE Transactions on Knowledge and Data Engineering, vol. 36, no. 4, pp. 1413–1430, Apr. 2024. doi: 10.1109/TKDE.2023.3310002

[4] Y. Liu, H. Li, A. Garcia-Duran, M. Niepert, D. Onoro-Rubio, and D. S. Rosenblum, "MMKG: Multi-Modal Knowledge Graphs," in Proc. ESWC, 2019, pp. 459-474. doi: 10.1007/978-3-030-21348-0_30

[5] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, "Learning Transferable Visual Models From Natural Language Supervision," in Proc. 38th Int. Conf. Mach. Learn. (ICML), vol. 139, 2021, pp. 8748-8763. doi: 10.48550/arXiv.2103.00020

[6] C. Jia, Y. Yang, Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. V. Le, Y. Sung, Z. Li, and T. Duerig, "Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision," in Proc. 38th Int. Conf. Mach. Learn. (ICML), vol. 139, 2021, pp. 4904-4916. doi: 10.48550/arXiv.2102.05918

[7] J. Yu, J. Li, Z. Yu, and Q. Huang, "Multimodal Transformer with Multi-View Visual Representation for Image Captioning," IEEE Trans. Circuits Syst. Video Technol., vol. 30, no. 12, pp. 4467-4480, Dec. 2020. doi: https://doi.org/10.1109/TCSVT.2019.2947482

[8] J. Li, D. Li, C. Xiong, and S. Hoi, "BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation," in Proc. 39th Int. Conf. Mach. Learn. (ICML), vol. 162, 2022, pp. 12888-12900. doi: 10.48550/arXiv.2201.12086

[9] L. Li, Z. Gan, Y. Cheng, and J. Liu, "Relation-Aware Graph Attention Network for Visual Question Answering," in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), Seoul, South Korea, 2019, pp. 10312–10321. doi: 10.1109/ICCV.2019.01041

[10] A. Khan, N. Chandran S, and D. R. Gangodkar, "Advanced Deep Learning Frameworks for Cross-Modal Information Retrieval: A Comprehensive Review of Techniques, Challenges, and Future Directions," Wiley Interdiscip. Rev.: Data Min. Knowl. Discov., vol. 16, no. 1, Art. no. e70055, 2026. doi: 10.1002/widm.70055

[11] R.-C. Tu, X.-L. Mao, B. Ma, Y. Hu, T. Yan, W. Wei, and H. Huang, "Deep Cross-Modal Hashing With Hashing Functions and Unified Hash Codes Jointly Learning," IEEE Transactions on Knowledge and Data Engineering, vol. 34, no. 2, pp. 560–572, Feb. 2022. doi: 10.1109/TKDE.2020.2987312

[12] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, "End-to-End Object Detection with Transformers," in Proc. Eur. Conf. Comput. Vis. (ECCV), Lecture Notes in Computer Science, vol. 12346, Cham: Springer, 2020, pp. 213–229. doi: 10.1007/978-3-030-58452-8_13

[13] Y. Lu, H. Rai, J. Chang, B. Knyazev, G. Yu, S. Shekhar, G. W. Taylor, and M. Volkovs, "Context-Aware Scene Graph Generation with Seq2Seq Transformers," in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), Oct. 2021, pp. 15911–15921. doi: 10.1109/ICCV48922.2021.01563

[14] J. Gu, H. Zhao, Z. Lin, S. Li, J. Cai, and M. Ling, "Scene Graph Generation with External Knowledge and Image Reconstruction," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2019, pp. 1969-1978. doi: 10.1109/CVPR.2019.00207

[15] J. Li, R. Selvaraju, A. Gotmare, S. Joty, C. Xiong, and S. Hoi, "Align before Fuse: Vision and Language Representation Learning with Momentum Distillation," in Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), vol. 34, 2021, pp. 9694-9705. doi: 10.48550/arXiv.2107.07651

[16] H. Tan and M. Bansal, "LXMERT: Learning Cross-Modality Encoder Representations from Transformers," in Proc. Conf. Empir. Methods Nat. Lang. Process. (EMNLP-IJCNLP), 2019, pp. 5100-5111. doi: 10.18653/v1/D19-1514

[17] W. Su, X. Zhu, Y. Cao, B. Li, L. Lu, F. Wei, and J. Dai, "VL-BERT: Pre-training of Generic Visual-Linguistic Representations," in Proc. Int. Conf. Learn. Represent. (ICLR), 2020. doi: 10.48550/arXiv.1908.08530

[18] K. Pham, K. Kafle, Z. Lin, Z. Ding, S. Cohen, "Learning to Predict Visual Attributes in the Wild," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2021, pp. 13202-13211. doi: 10.48550/arXiv.2106.09707

[19] K. Tang, Y. Niu, J. Huang, J. Shi, and H. Zhang, "Unbiased Scene Graph Generation from Biased Training," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2020, pp. 3716-3725. doi: 10.1109/CVPR42600.2020.00377

[20] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, "Microsoft COCO: Common Objects in Context," in Proc. Eur. Conf. Comput. Vis. (ECCV), 2014, pp. 740-755. doi: 10.1007/978-3-319-10602-1_48

[21] X. Zhu, W. Su, L. Lu, B. Li, X. Wang, and J. Dai, "Deformable DETR: Deformable Transformers for End-to-End Object Detection," in Proc. Int. Conf. Learn. Represent. (ICLR), 2021. [Online]. Available: https://openreview.net/forum?id=gZ9hCDWe6ke

[22] K. Desai and J. Johnson, "VirTex: Learning Visual Representations from Textual Annotations," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2021, pp. 11162-11173. doi: 10.1109/CVPR46437.2021.01101

[23] L. Yao et al., "FILIP: Fine-grained Interactive Language-Image Pre-Training," in Proc. Int. Conf. Learn. Represent. (ICLR), 2022. [Online]. Available: https://openreview.net/forum?id=cpDhcsEDC2

[24] Y. Li et al., "Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm," in Proc. Int. Conf. Learn. Represent. (ICLR), 2022. [Online]. Available: https://openreview.net/forum?id=zq1iJkNk3uN

[25] J. Li, D. Li, S. Savarese, and S. Hoi, "BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models," in Proc. 40th Int. Conf. Mach. Learn. (ICML), vol. 202, 2023, pp. 19730-19742. [Online]. Available: https://proceedings.mlr.press/v202/li23q.html

[26] B. Schroeder and S. Tripathi, "Structured Query-Based Image Retrieval Using Scene Graphs," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. Workshops (CVPRW), Jun. 2020, pp. 680–684. doi: 10.1109/CVPRW50498.2020.00098

[27] P. Zhang, X. Li, X. Hu, J. Yang, L. Zhang, L. Wang, Y. Choi, and J. Gao, "VinVL: Revisiting Visual Representations in Vision-Language Models," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2021, pp. 5575–5584. doi: 10.1109/CVPR46437.2021.00553