The Big Data Delusion: Even Infinite Information does not Guarantee Consistent Measurement

Abstract

Smart societies promise to enhance social systems by leveraging the constantly increasing stream of information around us. However, the possibility of optimizing various social indicators implicitly hinges on our ability to reliably reduce complex and abstract constructs (such as inter-personal trust) into numeric measures. While many scientists are aware of the issue of measurement error, there is widespread hope that access to more data will eventually render this issue irrelevant. This paper delves into the nature of measurement error under quasi-ideal conditions. We show mathematically and by employing simulations that single measurements fail to converge even when we have access to progressively more information. Then, by using data from the Social Capital Benchmark Surveys, we demonstrate how the addition of new information increases the dimensionality to the measured construct which further contributes to measurement divergence. In our conclusion, we stress that alternative evaluations to indexes and scores need to be evaluated for a fair society.