국내 커뮤니케이션 연구에서의 연구 재현성: 출판 편향과 p-해킹의 실증 연구

초록

The replication crisis has raised widespread concerns across scientific disciplines, leading to active discussions of low statistical power, selective reporting, and questionable research practices across the social sciences. However, these concerns have received limited attention in communication scholarship, particularly within the Korean academic context, despite replicability being a foundational criterion for scientific validity. This study aims to address this gap by quantitatively evaluating the potential for publication bias and p-hacking that may undermine research replicability in quantitative empirical studies published in Korean communication journals. To assess the current state of research practices, we conducted a comprehensive analysis of 1,086 articles published between 2020 and 2024 in the top 10 Korean communication journals, as ranked by the Korea Citation Index (KCI). From this sample, 405 studies containing statistical significance tests were selected, yielding a total of 6,481 test statistics for analysis. To evaluate potential biases in these reported results, we applied z-curve analysis—a statistical tool designed to detect publication bias and p-hacking by modeling the distribution of significant test outcomes. Using this method, we estimated five key parameters: the Observed Discovery Rate (ODR), Expected Discovery Rate (EDR), Expected Replication Rate (ERR), False Discovery Rate (FDR), and the File Drawer Ratio (FDRatio). The analysis revealed a substantial gap between the proportion of statistically significant results reported (ODR = 59.1%) and the proportion expected based on statistical power (EDR = 23.5%). This discrepancy indicates a likely inflation of significant findings due to selective reporting or suppression of null results. The estimated ERR was 74.6%, while the FDR indicated that 17.1% of significant results might be false positives. Furthermore, the implied FDRatio suggested that, on average, three nonsignificant findings may have been omitted for every significant one published. These results suggest the presence of questionable research practices—such as publication bias and p-hacking—within Korean communication research, and the magnitude of these distortions is nontrivial. In some journals, the extent of bias was found to be even more pronounced. Compared to findings from prior meta-scientific investigations in other fields, the patterns observed in this study raise concerns about the reliability of published results in Korean communication scholarship. These findings point to structural challenges in Korean communication research, including the widespread reliance on p-value–driven reporting, the lack of pre-registration, and limited attention to statistical power. Enhancing research replicability and credibility will require coordinated efforts toward institutional reform, methodological transparency, and a stronger culture of openness and accountability in scientific practice. By offering empirical evidence on the existence of p-hacking, the presence of publication bias, and their substantive impact and magnitude, this study contributes to the growing body of meta-scientific research and provides practical insights.

키워드