Efficient Chain-of-Thought Guided Instruction Tuning for Graph Representation Learning
DOI:
https://doi.org/10.29207/joseit.v3i3.8453Keywords:
graph representation learning, chain-of-thought, graph encoder, cross-modal alignment, instruction tuningAbstract
Large language models (LLMs) reason well over natural language but do not naturally understand graph structure, and directly applying them to graph data runs into two persistent obstacles: a wide gap between the graph and text modalities, and the difficulty of encoding structural information in a way an LLM can use. Existing graph-instruction-tuning methods typically close this gap with simple linear projections and question-answer pairs that carry no explicit reasoning trace, which leaves cross-modal alignment shallow and drives up fine-tuning cost. This paper proposes GraphCoT, a graph representation learning method built on efficient chain-of-thought (CoT) fine-tuning. Its core is a high-quality CoT distillation mechanism in which a strong teacher model generates instruction data containing explicit intermediate reasoning paths, which explicitly teaches a student model to reason in multiple steps from only a small amount of training data. GraphCoT further introduces a graph-text alignment module that maps graph representations into the language embedding space, and a lightweight two-stage training strategy that aligns modalities before task-specific fine-tuning, which keeps training cost low while preserving both reasoning quality and transferability. Node classification and link prediction experiments on multiple benchmark datasets show that GraphCoT outperforms mainstream graph learning methods in accuracy and generalization, confirming the effectiveness of the CoT distillation mechanism and the alignment strategy for graph representation learning.
References
XU X, ZHANG Y F, ZHOU F, et al. Improving multimodal social media popularity prediction via selective retrieval knowledge augmentation. Proceedings of the AAAI Conference on Artificial Intelligence, 2025, 39(1): 932-940.
GILMER J, SCHOENHOLZ S S, RILEY P F, et al. Neural message passing for quantum chemistry. Proceedings of the 34th International Conference on Machine Learning, 2017: 1263-1272.
WANG S Y, SUI Y D, WANG C, et al. Unleashing the power of knowledge graph for recommendation via invariant learning. Proceedings of the ACM Web Conference 2024. New York: ACM, 2024: 3745-3755.
CHEN Y L, WU T X, LIU Y C, et al. Uncertain knowledge graph completion with rule mining. Proceedings of the 21st International Conference on Web Information Systems and Applications. Singapore: Springer, 2024: 100-112.
SUN S Y, REN Y X, CHEN J H, et al. Large language models as topological structure enhancers for text-attributed graphs. Proceedings of the 30th International Conference on Database Systems for Advanced Applications. Singapore: Springer, 2026: 106-122.
CHEN Z K, MAO H T, WEN H Z, et al. Label-free node classification on graphs with large language models (LLMs). arXiv:2310.04668, 2023.
YE R S, ZHANG C Q, WANG R H, et al. Language is all a graph needs. Findings of the Association for Computational Linguistics: EACL 2024. Stroudsburg: ACL, 2024: 1955-1973.
LUO Z H, SONG X R, HUANG H, et al. GraphInstruct: empowering large language models with graph understanding and reasoning capability. arXiv:2403.04483, 2024.
WANG J N, WU J D, HOU Y P, et al. InstructGraph: boosting large language models via graph-centric instruction tuning and preference alignment. Findings of the Association for Computational Linguistics: ACL 2024. Stroudsburg: ACL, 2024: 13492-13510.
TANG J B, YANG Y H, WEI W, et al. GraphGPT: graph instruction tuning for large language models. Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. New York: ACM, 2024: 491-500.
KOJIMA T, GU S S, REID M, et al. Large language models are zero-shot reasoners. Proceedings of the 36th International Conference on Neural Information Processing Systems, 2022: 22199-22213.
WEI J, TAY Y, BOMMASANI R, et al. Emergent abilities of large language models. arXiv:2206.07682, 2022.
MIN S, LYU X X, HOLTZMAN A, et al. Rethinking the role of demonstrations: what makes in-context learning work? Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Stroudsburg: ACL, 2022: 11048-11064.
KUMAR S, MENDKE S, RAHMAN K L A, et al. Enhancing zero-shot chain of thought prompting via uncertainty-guided strategy selection. Proceedings of the 31st International Conference on Computational Linguistics. Stroudsburg: ACL, 2025: 2003-2025.
OUYANG L, WU J, XU J, et al. Training language models to follow instructions with human feedback. Proceedings of the 36th International Conference on Neural Information Processing Systems, 2022: 27730-27744.
BAI Y T, KADAVATH S, KUNDU S, et al. Constitutional AI: harmlessness from AI feedback. arXiv:2212.08073, 2022.
TOUVRON H, LAVRIL T, IZACARD G, et al. LLaMA: open and efficient foundation language models. arXiv:2302.13971, 2023.
TOUVRON H, MARTIN L, STONE K, et al. Llama 2: open foundation and fine-tuned chat models. arXiv:2307.09288, 2023.
DEEPSEEK-AI, LIU A X, FENG B, et al. DeepSeek-V3 technical report. arXiv:2412.19437, 2024.
XU J, GUO Z F, HE J Z, et al. Qwen2.5-Omni technical report. arXiv:2503.20215, 2025.
WEI J, WANG X Z, SCHUURMANS D, et al. Chain-of-thought prompting elicits reasoning in large language models. Proceedings of the 36th International Conference on Neural Information Processing Systems, 2022: 24824-24837.
MAGISTER L C, MALLINSON J, ADAMEK J, et al. Teaching small language models to reason. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. Stroudsburg: ACL, 2023: 1773-1781.
HSIEH C Y, LI C L, YEH C K, et al. Distilling step-by-step! Outperforming larger language models with less training data and smaller model sizes. Findings of the Association for Computational Linguistics: ACL 2023. Stroudsburg: ACL, 2023: 8003-8017.
CHEN X, HUANG H X, GAO Y J, et al. Learning to maximize mutual information for chain-of-thought distillation. Findings of the Association for Computational Linguistics: ACL 2024. Stroudsburg: ACL, 2024: 6857-6868.
FENG K T, LI C S, ZHANG X L, et al. Keypoint-based progressive chain-of-thought distillation for LLMs. Proceedings of the 41st International Conference on Machine Learning, 2024: 13241-13255.
WANG P F, WANG Z Y, LI Z, et al. SCOTT: self-consistent chain-of-thought distillation. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. Stroudsburg: ACL, 2023: 5546-5558.
LI L H, HESSEL J, YU Y, et al. Symbolic chain-of-thought distillation: small models can also “think” step-by-step. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. Stroudsburg: ACL, 2023: 2665-2679.
ZHUANG X W, ZHU Z H, WANG Z C, et al. UniCoTT: a unified framework for structural chain-of-thought distillation. Proceedings of the 13th International Conference on Learning Representations, 2025.
LEE Y J, LI C Y, LIU H T, et al. Visual instruction tuning. Advances in Neural Information Processing Systems 36, 2023: 34892-34916.
ZHU D Y, CHEN J, SHEN X Q, et al. MiniGPT-4: enhancing vision-language understanding with advanced large language models. arXiv:2304.10592, 2023.
SHU D, CHEN T L, JIN M Y, et al. Knowledge graph large language model (KG-LLM) for link prediction. Proceedings of the 22nd Asian Conference on Machine Learning, 2025: 143-158.
MA J, GAO Z T, CHAI Q, et al. Debate on graph: a flexible and reliable reasoning framework for large language models. Proceedings of the AAAI Conference on Artificial Intelligence, 2025, 39(23): 24768-24776.
MA S J, XU C J, JIANG X H, et al. Think-on-graph 2.0: deep and faithful large language model reasoning with knowledge-guided retrieval augmented generation. arXiv:2407.10805, 2024.
LIU B, ZHANG J H, LIN F Q, et al. Filter-then-generate: large language models with structure-text adapter for knowledge graph completion. Proceedings of the 31st International Conference on Computational Linguistics. Stroudsburg: ACL, 2025: 11181-11195.
HU S X, ZOU G B, YANG S, et al. Large language model meets graph neural network in knowledge distillation. Proceedings of the AAAI Conference on Artificial Intelligence, 2025, 39(16): 17295-17304.
GAO H, ZHANG C H, WU F G, et al. Bootstrapping heterogeneous graph representation learning via large language models: a generalized approach. Proceedings of the AAAI Conference on Artificial Intelligence, 2025, 39(16): 16717-16726.
DING J, ZHENG Z, LIN B, et al. MSG-LLM: a multi-scale interactive framework for graph-enhanced large language models. Proceedings of the 31st International Conference on Computational Linguistics. Stroudsburg: ACL, 2025: 9687-9700.
DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Stroudsburg: ACL, 2019: 4171-4186.
HU W, FEY M, ZITNIK M, et al. Open graph benchmark: datasets for machine learning on graphs. Advances in Neural Information Processing Systems 33, 2020: 22118-22133.
HE X X, BRESSON X, LAURENT T, et al. Explanations as features: LLM-based features for text-attributed graphs. arXiv:2305.19523, 2023.
KIPF T N, WELLING M. Semi-supervised classification with graph convolutional networks. Proceedings of the 5th International Conference on Learning Representations, 2017.
WEN Z H, FANG Y. Augmenting low-resource text classification with graph-grounded pre-training and prompting. Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. New York: ACM, 2023: 506-516.
HAMILTON W L, YING R, LESKOVEC J. Inductive representation learning on large graphs. Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017: 1025-1035.
VELICKOVIC P, CUCURULL G, CASANOVA A, et al. Graph attention networks. Proceedings of the 6th International Conference on Learning Representations, 2018.
LI G, MÜLLER M, GHANEM B, et al. Training graph neural networks with 1000 layers. Proceedings of the 38th International Conference on Machine Learning, 2021: 6437-6449.
VELICKOVIC P, FEDUS W, HAMILTON W L, et al. Deep graph infomax. Proceedings of the 7th International Conference on Learning Representations, 2019.
WU Q T, YAN J C, YANG C X. Geometric knowledge distillation: topology compression for graph neural networks. Advances in Neural Information Processing Systems 35, 2022: 29761-29775.
ZHANG S, LIU Y, SUN Y, et al. Graph-less neural networks: teaching old MLPs new tricks via distillation. Proceedings of the 10th International Conference on Learning Representations, 2022.
LI Z N, WIPF D P, WU Q T, et al. NodeFormer: a scalable graph structure learning transformer for node classification. Advances in Neural Information Processing Systems 35, 2022: 27387-27401.
WU Q T, YANG C X, ZHAO W T, et al. DIFFormer: scalable (graph) transformers induced by energy constrained diffusion. arXiv:2301.09474, 2023.
Downloads
Published
Issue
Section
License
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under Creative Commons Attribution 4.0 International License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (Refer to The Effect of Open Access).
