Bad Mood Rising? Assessing Scalar Invariance Violations with Comparative Democratic Mood Data

Abstract

The advent of nearly global estimates of democratic mood represents a genuine cause for optimism around identifying the linkages between public opinion and democracy. This has recently been boosted by claims that novel approaches to measuring latent democratic mood with hierarchical IRT measurement models overcome cross-national differential item functioning (DIF) and, in turn, the biases from violations of scalar invariance – fundamental challenges that typically foil the comparability of country-level estimates of latent quantities. Focusing specifically on the issue of scalar invariance, we show mathematically and with statistical simulations that no commonly used latent variable modeling framework – including hierarchical IRT – is immune to DIF. While some modeling approaches can fully accommodate measurement noninvariance which is completely random between countries and across items, they falter as soon as DIF has a directional bias, i.e., if survey items are systematically harder (or easier) to answer across geographic or temporal units. Equipped with democratic support data from Latin America, we find suggestive evidence that such directional DIF is far more prevalent than random measurement noninvariance. We conclude with a number of practical recommendations to mitigate DIF and scalar invariance in their own research.