The Influence of NPMI and TF-IDF-Based Automatic Stopword Generation on Semantic Consistency

초록

PURPOSE This study optimized stopword removal to enhance topic modeling performance. We propose an objective method combining normalized pointwise mutual information (NPMI) with median-based term frequency–inverse document frequency (TF–IDF) to automatically generate stopwords. METHODS Using text data from 443 research papers on “Taekwondo sparring,” we selected stopword candidates based on NPMI and identified 30 words with the lowest TF–IDF scores. We examined the impact of removing 1–30 stopwords on u_mass coherence scores. RESULTS The NPMI–TF–IDF method significantly improved coherence (R2 = .456; p < .001). However, excessive removal led to diminishing returns, with the optimal coherence score (−11.442) achieved at 200 stopwords. In contrast, manually selected stopwords yielded a lower coherence score (−16.001). The findings indicate that integrating TF– IDF with NPMI effectively preserves meaningful words and outperforms PMI2 and PMI3 approaches. CONCLUSIONS Manual stopword selection can reduce reproducibility. Optimizing stopword removal based on domain-specific characteristics is essential. Future research should validate this method across diverse fields to establish a more generalizable standard. © 2025, Korea Institute of Sport Science. All rights reserved.

키워드