Whose data can we trust: How meta-predictions can be used to uncover credible respondents in survey data

Autoři: Sonja Radas aff001;  Drazen Prelec aff002
Působiště autorů: Institute of Economics, Zagreb, Croatia aff001;  Massachusetts Institute of Technology, Cambridge, Massachusetts, United States of America aff002
Vyšlo v časopise: PLoS ONE 14(12)
Kategorie: Research Article
doi: 10.1371/journal.pone.0225432


Many areas of economics use subjective data, although it had been known to present problems regarding its reliability. To improve data quality, researchers may use scoring rules that reward respondents so that it is most profitable for them to tell the truth. However, if the subjects are not well informed about the topic or if they do not pay sufficient attention, they will produce data that could not be dependably used for decision-making even though subjects gave their honest answer. In this paper we show how meta-predictions (respondents’ predictions about choices of others) can be used for identification of respondents who produce dependable data. We use purchase intention survey, a popular method to elicit early adoption forecasts for a new concept, as a test bed for our approach. We present results from three online experiments, demonstrating that corrected purchase intentions are closer to the real outcomes.

Klíčová slova:

Algorithms – Experimental economics – Game theory – Questionnaires – Research validity – Surveys – Smart materials


1. Prelec D. A Bayesian Truth Serum for Subjective Data. Science, 2004; 306: 462–466. doi: 10.1126/science.1102081 15486294

2. Gneiting T, Raftery AE. Strictly Proper Scoring Rules, Prediction, and Estimation, Journal of the American Statistical Association, 2007; 102(477): 359–378

3. Balasubramanian S, Gupta S, Kamakura W, Wedel M. Modeling large data sets in marketing. Statistica Neerlandica, 1998, 52(3): 303.

4. Kahneman D, Snell J. Predicting a changing task: Do people know what they will like? Journal of Behavioral Decision Making, 1992; 5:187–200.

5. Chintagunta P, Lee J. A pre-diffusion growth model of intentions and purchase. Journal of the Academy of Marketing Science, 2012, 40(1): 137–154.

6. Morwitz VG, Steckel JH, Gupta A. When do purchase intentions predict sales? International Journal of Forecasting. 2007; 23(3): 347–364.

7. Reichheld FF. The one number you need to grow. Harvard Business Review, 2003; 81(12): 46–54. 14712543

8. Wansink B, Ray ML. Advertising Strategies to Increase Usage Frequency Journal of Marketing, 1996; 60(1): 31–46.

9. Bemmaor AC. Predicting behavior from intention-to-buy measures: The parametric case, Journal of Marketing Research, 1995, 32:176–191

10. Hsiao C, Sun B, Morwitz VG. The role of stated intentions in new product purchase forecasting, in Advances in Econometrics. 2002: 11–28.

11. Juster FT. Consumer buying intentions and purchase probability. Occasional Paper 99, National Bureau of Economic Research, Colombia University Press; 1996.

12. Kalwani MU, Silk AJ. On the reliability and predictive validity of purchase intention measures, Marketing Science, 1982;(1): 243–286.

13. Manski C. Nonparametric Bounds on Treatment Effects. The American Economic Review, 1990; 80(2): 319–323.

14. Mittal V, Kamakura WA. Satisfaction, repurchase intent, and repurchase behavior: Investigating the moderating effect of customer characteristics. Journal of Marketing Research, 2001; 38(1): 131–142.

15. Morrison DG. Purchase intentions and purchase behavior. Journal of Marketing, 1979; 43: 65–74

16. Morwitz VG, Schmittlein D. Using segmentation to improve sales forecasts based on purchase intent: Which “intenders” actually buy? Journal of Marketing Research, 1992; 29: 391–405

17. Sun B, Morwitz VG. Stated intentions and purchase behavior: A unified model. International Journal Of Research In Marketing, 2010; 27(4): 356–366.

18. Helzer E, Dunning D. Why and when peer prediction is superior to self-prediction: The weight given to future aspiration versus past achievement. Journal of personality and social psychology. 2012; 103 (1): 38–53. doi: 10.1037/a0028124 22506647

19. Rothschild D, Wolfers J. Forecasting Elections: Voter Intentions versus Expectations, NBER Working Paper.2013. http://users.nber.org/~jwolfers/Papers/VoterExpectations.pdf

20. Galesic M, de Bruin WB, Dumas M, Kapteyn A, Darling JE, Meijer E. Asking about social circles improves election predictions. Nature Human Behaviour. 2018; 2: 187–193.

21. Court D, Gillen B, McKenzie J, Plott CR. Two information aggregation mechanisms for predicting the opening weekend box office revenues of films: Box office Prophecy and Guess of Guesses. Economic Theory. 2018; 65(1): 25–54.

22. Prelec D, Seung HS, McCoy J. A solution to the single-question crowd wisdom problem. Nature. 2017; 541: 532–535. doi: 10.1038/nature21054 28128245

23. Palley AB, Soll JB. Extracting the Wisdom of Crowds When Information is Shared. Management Science. 2018 forthcoming.

24. Weaver R, Prelec D. Creating truth-telling incentives with the Bayesian truth serum. Journal of Marketing Research. 2013; 50(3): 289–302.

25. Wedel M, DeSarbo WS. A mixture likelihood approach for generalized linear models, Journal of Classification. 1995; 12: 21–55.

26. Wedel M, Kamakura WA. Factor Analysis with Mixed Observed and Latent Variables in the Exponential Family. Psychometrika 2001; 66(4): 515–30.

27. Leisch F. FlexMix: A General Framework for Finite Mixture Models and Latent Class Regression in R. Journal of Statistical Software, 2004;11(8): 1–18.

Článek vyšel v časopise


2019 Číslo 12