From Lexicons to Large Language Models: A Holistic Evaluation of Psychometric Text Analysis in Social Science Research

Research Spotlight Abstract Extracting psychological insights from text is vital for modern analytics, yet organizations often rely on analysis tools that are either biased and simplistic or prohibitively expensive to build. Our research demonstrates that Large Language Models (LLMs) offer a superior alternative. They match the accuracy of specialized artificial intelligence (AI), while significantly reducing costs and technical barriers. Crucially for policy considerations, we find LLMs are statistically fairer than traditional methods. In our tests, they reduced racial and gender bias by up to 60%. Beyond assessing performance, we introduce a practical technique called “cognitive-affective prompting.” By instructing the AI to adopt specific human strengths, such as using “superior reasoning” for complex tasks or “emotional intelligence” for sentiment analysis, practitioners can boost accuracy by over 10%. To facilitate adoption, we provide a user-friendly “cookbook” to help nonexperts apply these findings immediately. For policymakers and business leaders, this research validates LLMs as a robust, consistent, and equitable standard for analyzing human behavior at scale.

• 저널Information Systems Research
• 토픽Mental Health via Writing · Social Psychology
• DOI10.1287/isre.2024.1143
• 저자Reza Mousavi, Brent Kitchens, Abbie Griffith Oliver, Ahmed Abbasi