Large Language Model-Based Extraction of Logic Rules from Technical Standards for Automatic Compliance Checking
Abstract
In this research, we design logic rules as a representation of technical standards documents related to ship design, which will be used in automatic compliance checking. We present a novel design of logic rules based on a general pattern of technical standards’ clauses that can be produced automatically from text using a large language model (LLM). We also present a method to extract said logic rules from text. First, we design data structures to represent the technical standards and logic rules used to process the data. Second, the representation of technical standards is produced manually and tested to ensure that it can give the same conclusion as human judgment regarding compliance. Third, a variation of prompting methods, namely pipeline method and few-shot prompting, is given to LLM to instruct it to extract logic rules from text following the design. Evaluation against the logic rules produced shows that the pipeline method gives an accuracy score of 0.57, a precision of 0.49, and a recall of 0.62. On the other hand, logic rules extracted using few-shot prompting have an accuracy score of 0.33, precision of 0.43, and recall of 0.5. These results show that LLM is able to extract a logic rule representation of technical standards. Furthermore, the representation resulting from the prompting technique that utilizes the pipeline method has a better performance compared to the representation resulting from few-shot prompting.
Downloads
References
“Rules for Classification: Ships - Part 1 General Regulations - Chapter 1 General Regulations,” Jul. 2023
L. Jiang, J. Shi, and C. Wang, “Multi-ontology fusion and rule development to facilitate automated code compliance checking using BIM and rule-based reasoning,” Advanced Engineering Informatics, vol. 51, Jan. 2022, doi: 10.1016/j.aei.2021.101449.
W3C SPARQL Working Group, “SPARQL 1.1 Overview.” Accessed: Jun. 08, 2024. [Online]. Available: https://www.w3.org/TR/2013/REC-sparql11-overview-20130321/
P. Patlakas, I. Christovasilis, L. Riparbelli, F. K. Cheung, and E. Vakaj, “Semantic web-based automated compliance checking with integration of Finite Element analysis,” Advanced Engineering Informatics, vol. 61, Aug. 2024, doi: 10.1016/j.aei.2024.102448.
X. Zhu, H. Li, and T. Su, “Autonomous complex knowledge mining and graph representation through natural language processing and transfer learning,” Autom Constr, vol. 155, Nov. 2023, doi: 10.1016/j.autcon.2023.105074.
M. Yang et al., “Semi-automatic representation of design code based on knowledge graph for automated compliance checking,” Comput Ind, vol. 150, Sep. 2023, doi: 10.1016/j.compind.2023.103945.
I. Fitkau and T. Hartmann, “An ontology-based approach of automatic compliance checking for structural fire safety requirements,” Advanced Engineering Informatics, vol. 59, Jan. 2024, doi: 10.1016/j.aei.2023.102314.
D. Guo, E. Onstein, and A. D. La Rosa, “A Semantic Approach for Automated Rule Compliance Checking in Construction Industry,” IEEE Access, vol. 9, pp. 129648–129660, 2021, doi: 10.1109/ACCESS.2021.3108226.
X. Zhao, L. Huang, Z. Sun, X. Fan, and M. Zhang, “Compliance Checking on Topological Spatial Relationships of Building Elements Based on Building Information Models and Ontology,” Sustainability (Switzerland), vol. 15, no. 14, Jul. 2023, doi: 10.3390/su151410901.
X. Xue and J. Zhang, “Regulatory information transformation ruleset expansion to support automated building code compliance checking,” Autom Constr, vol. 138, Jun. 2022, doi: 10.1016/j.autcon.2022.104230.
Q. Ren et al., “Automatic quality compliance checking in concrete dam construction: Integrating rule syntax parsing and semantic distance,” Advanced Engineering Informatics, vol. 60, Apr. 2024, doi: 10.1016/j.aei.2024.102409.
S. Liu, B. Zhao, R. Guo, G. Meng, F. Zhang, and M. Zhang, “Have you been properly notified? automatic compliance analysis of privacy policy text with GDPR article 13,” in The Web Conference 2021 - Proceedings of the World Wide Web Conference, WWW 2021, Association for Computing Machinery, Inc, Apr. 2021, pp. 2154–2164. doi: 10.1145/3442381.3450022.
J. Devlin, M.-W. Chang, K. Lee, K. T. Google, and A. I. Language, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” in Proceedings of NAACL-HLT 2019, 2019, pp. 4171–4186. doi: 10.18653/v1/N19-1423.
A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, “Improving Language Understanding by Generative Pre-Training,” 2018, Accessed: Jan. 06, 2025. [Online]. Available: https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf
H. Touvron et al., “LLaMA: Open and Efficient Foundation Language Models,” 2023, doi: https://doi.org/10.48550/arXiv.2302.13971.
A. Chowdhery et al., “PaLM: Scaling Language Modeling with Pathways,” Apr. 2022, doi: https://doi.org/10.48550/arXiv.2204.02311.
S. Carta, A. Giuliani, L. Piano, A. S. Podda, L. Pompianu, and S. G. Tiddia, “Iterative Zero-Shot LLM Prompting for Knowledge Graph Construction,” ArXiv, Jul. 2023, doi: https://doi.org/10.48550/arXiv.2307.01128.
J. H. Caufield et al., “Structured prompt interrogation and recursive extraction of semantics (SPIRES): A method for populating knowledge bases using zero-shot learning,” Bioinformatics, Feb. 2024, doi: 10.1093/bioinformatics/btae104.
Z. Bi et al., “CodeKGC: Code Language Model for Generative Knowledge Graph Construction,” ACM Transactions on Asian and Low-Resource Language Information Processing, Feb. 2024, doi: 10.1145/3641850.
I. Bratko, Prolog Programming for Artificial Intelligence, 4th ed. Harlow: Addison Wesley, 2012.
J. K. Lee, K. Cho, H. Choi, S. Choi, S. Kim, and S. H. Cha, “High-level implementable methods for automated building code compliance checking,” Developments in the Built Environment, vol. 15, Oct. 2023, doi: 10.1016/j.dibe.2023.100174.
S. Ramanauskaitė, A. Shein, A. Čenys, and J. Rastenis, “Security Ontology Structure for Formalization of Security Document Knowledge,” Electronics (Switzerland), vol. 11, no. 7, Apr. 2022, doi: 10.3390/electronics11071103.
T. B. Brown et al., “Language Models are Few-Shot Learners,” in 34th Conference on Neural Information Processing Systems (NeurIPS 2020), 2020. [Online]. Available: https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf
J. Wei et al., “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models,” in Proceedings of the 36th International Conference on Neural Information Processing Systems, 2022.
L. Zhong, Z. Wang, and J. Shang, “Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step by Step,” Findings of the Association for Computational Linguistics: ACL 2024, Aug. 2024, doi: 10.18653/v1/2024.findings-acl.49.
D. Huang, J. M. Zhang, M. Luck, Q. Bu, Y. Qing, and H. Cui, “AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation,” Dec. 2023, doi: https://doi.org/10.48550/arXiv.2312.13010.
Biro Klasifikasi Indonesia, “Rules for Electrical Installations Consolidated Edition,” Part 1, Vol. IV, 2024 [Online]. Available: www.bki.co.id
Copyright (c) 2025 Jurnal RESTI (Rekayasa Sistem dan Teknologi Informasi)
This work is licensed under a Creative Commons Attribution 4.0 International License.
Copyright in each article belongs to the author
- The author acknowledges that the RESTI Journal (System Engineering and Information Technology) is the first publisher to publish with a license Creative Commons Attribution 4.0 International License.
- Authors can enter writing separately, arrange the non-exclusive distribution of manuscripts that have been published in this journal into other versions (eg sent to the author's institutional repository, publication in a book, etc.), by acknowledging that the manuscript has been published for the first time in the RESTI (Rekayasa Sistem dan Teknologi Informasi) journal ;
