p -Hacking and Publication Bias in Design-Based Causal Studies in Information Systems

Design-based causal methods such as difference in differences, instrumental variables, randomized controlled trials, regression discontinuity, and synthetic control have become the foundation of evidence-based decision making in information systems (IS). Their value, however, depends on the credibility of the published evidence. Analyzing 7,516 hypothesis tests from 558 articles published in seven leading IS journals between 2000 and 2023, we identify systematic clustering of statistical results immediately above the conventional 5% significance threshold, particularly for difference in differences, instrumental variables, and randomized controlled trials. These patterns suggest that publication incentives and researcher degrees of freedom may distort the evidence base used by managers and policymakers. We translate these findings into practical recommendations for researchers, reviewers, and editors. Specifically, we advocate greater transparency in research design and reporting, stronger requirements for data and code availability, broader acceptance of well-executed null findings, routine sensitivity analyses, and method-specific reporting standards for causal studies. We also introduce an IS-calibrated local false discovery rate that helps readers assess the credibility of reported causal estimates. By strengthening evidence quality rather than discouraging causal research, these recommendations aim to improve the reliability, reproducibility, and policy relevance of empirical findings that increasingly guide organizational investment, digital transformation, platform governance, and public policy decisions.

• 저널Information Systems Research
• 토픽Advanced Causal Inference Techniques · Statistics and Probability
• DOI10.1287/isre.2024.1365
• 저자Mike Nguyen, Minh Anh Nguyen