BERTopic-Driven Identification of Emerging Technology Topics: A Multi-Source Framework with Empirical Validation in New Energy Vehicles
DOI:
https://doi.org/10.29207/joseit.v4i2.7169Keywords:
topic modeling, BERTopic, technology intelligence, patent analysis, semantic analysisAbstract
This study addresses the critical challenge of accurately detecting emerging technology topics from large-scale heterogeneous data sources. The methodology encompasses four sequential stages: (1) multi-source data acquisition from academic papers, patents, policies, and technical reports; (2) BERTopic-based topic modeling utilizing BERT embeddings and c-TF-IDF for enhanced semantic representation; (3) topic consolidation through cosine similarity analysis of topic vectors; and (4) emerging topic identification via a weighted evaluation system incorporating novelty, growth, continuity, and impact dimensions. Applied to the new energy vehicle domain using data from 2010-2022, the framework successfully identified 16 candidate emerging technology topics through analysis of 27,058 academic papers and 54,572 patents. Validation results indicate that 12 of the 16 identified topics (75% accuracy) align with technological priorities outlined in government policies and industry reports. The method effectively captures cross-domain technological convergence, with four common topics identified between academic and patent datasets, primarily concentrated in battery technology domains.
References
D. Rotolo, D. Hicks, and B. R. Martin, “What is an emerging technology?,” Res Policy, vol. 44, no. 10, 2015, doi: 10.1016/j.respol.2015.06.006.
S. Cozzens et al., “Emerging technologies: Quantitative identification and measurement,” Technol Anal Strateg Manag, vol. 22, no. 3, 2010, doi: 10.1080/09537321003647396.
D. F. Reding and J. Eaton, “Science and Technology Trends 2020-2040,” Nato Science and Technology Organisation, 2020.
Y. Zhou, F. Dong, D. Kong, and Y. Liu, “Unfolding the convergence process of scientific knowledge for the early identification of emerging technologies,” Technol Forecast Soc Change, vol. 144, 2019, doi: 10.1016/j.techfore.2019.03.014.
J. Joung and K. Kim, “Monitoring emerging technologies for technology planning using technical keyword based analysis from patent data,” Technol Forecast Soc Change, vol. 114, 2017, doi: 10.1016/j.techfore.2016.08.020.
B. Wang, Y. Liu, Y. Zhou, and Z. Wen, “Emerging nanogenerator technology in China: A review and forecast using integrating bibliometrics, patent analysis and technology roadmapping methods,” Nano Energy, vol. 46, 2018, doi: 10.1016/j.nanoen.2018.02.020.
S. H. Chang, “Technical trends of artificial intelligence in standard-essential patents,” Data Technologies and Applications, vol. 55, no. 1, 2021, doi: 10.1108/DTA-10-2019-0178.
A. J. C. Trappey, P. P. J. Chen, C. V. Trappey, and L. Ma, “A machine learning approach for solar power technology review and patent evolution analysis,” Applied Sciences (Switzerland), vol. 9, no. 7, 2019, doi: 10.3390/app9071478.
T. Verbeij, I. Beyens, D. Trilling, and P. M. Valkenburg, “Happiness and Sadness in Adolescents’ Instagram Direct Messaging: A Neural Topic Modeling Approach,” Social Media and Society, vol. 10, no. 1, 2024, doi: 10.1177/20563051241229655.
A. L. Porter, J. Garner, S. F. Carley, and N. C. Newman, “Emergence scoring to identify frontier R&D topics and key players,” Technol Forecast Soc Change, vol. 146, 2019, doi: 10.1016/j.techfore.2018.04.016.
J. Devlin, M. W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in NAACL HLT 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference, 2019.
S. Samsir, R. S. Saragih, S. Subagio, R. Aditiya, and R. Watrianthos, “BERTopic Modeling of Natural Language Processing Abstracts: Thematic Structure and Trajectory,” Jurnal Media Informatika Budidarma, vol. 7, no. 3, Jul. 2023.
Maarten Grootendorst, “BERTopic: Leveraging BERT and c-TF-IDF to create easily interpretable topics,” Zenodo, 2020.
J. Wang, Z. Zhang, L. Feng, K. Y. Lin, and P. Liu, “Development of technology opportunity analysis based on technology landscape by extending technology elements with BERT and TRIZ,” Technol Forecast Soc Change, vol. 191, 2023, doi: 10.1016/j.techfore.2023.122481.
L. Huang, X. Chen, X. Ni, J. Liu, X. Cao, and C. Wang, “Tracking the dynamics of co-word networks for emerging topic identification,” Technol Forecast Soc Change, vol. 170, 2021, doi: 10.1016/j.techfore.2021.120944.
L. McInnes, J. Healy, and S. Astels, “hdbscan: Hierarchical density based clustering,” The Journal of Open Source Software, vol. 2, no. 11, 2017, doi: 10.21105/joss.00205.
H. Small, “Co‐citation in the scientific literature: A new measure of the relationship between two documents,” Journal of the American Society for Information Science, vol. 24, no. 4, 1973, doi: 10.1002/asi.4630240406.
M. E. J. Newman, “Coauthorship networks and patterns of scientific collaboration,” Proc Natl Acad Sci U S A, vol. 101, no. SUPPL. 1, 2004, doi: 10.1073/pnas.0307545100.
L. McInnes, J. Healy, N. Saul, and L. Großberger, “UMAP: Uniform Manifold Approximation and Projection,” J Open Source Softw, vol. 3, no. 29, 2018, doi: 10.21105/joss.00861.
W. A. Prabowo and F. Azizah, “Sentiment Analysis for Detecting Cyberbullying Using TF-IDF and SVM,” Jurnal RESTI (Rekayasa Sistem dan Teknologi Informasi), vol. 4, no. 6, pp. 1142 – 1148–1142 – 1148, 2020, doi: 10.29207/resti.v4i6.2753.
S. W. Kim and J. M. Gil, “Research paper classification systems based on TF-IDF and LDA schemes,” Human-centric Computing and Information Sciences, vol. 9, no. 1, 2019, doi: 10.1186/s13673-019-0192-7.
Y. Cho et al., “Identifying technology opportunities for electric motors of railway vehicles with patent analysis,” Sustainability (Switzerland), vol. 13, no. 5, 2021, doi: 10.3390/su13052424.
D. Grimshaw, “Forecasting and management of technology,” The Journal of Strategic Information Systems, vol. 1, no. 1, 1991, doi: 10.1016/0963-8687(91)90009-8.
Downloads
Published
Issue
Section
License
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under Creative Commons Attribution 4.0 International License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (Refer to The Effect of Open Access).
