Measuring the predictability of life outcomes with a scientific mass collaboration
Abstract
Significance Hundreds of researchers attempted to predict six life outcomes, such as a child’s grade point average and whether a family would be evicted from their home. These researchers used machine-learning methods optimized for prediction, and they drew on a vast dataset that was painstakingly collected by social scientists over 15 y. However, no one made very accurate predictions. For policymakers considering using predictive models in settings such as criminal justice and child-protective services, these results raise a number of concerns. Additionally, researchers must reconcile the idea that they understand life trajectories with the fact that none of the predictions were very accurate. How predictable are life trajectories? We investigated this question with a scientific mass collaboration using the common task method; 160 teams built predictive models for six life outcomes using data from the Fragile Families and Child Wellbeing Study, a high-quality birth cohort study. Despite using a rich dataset and applying machine-learning methods optimized for prediction, the best predictions were not very accurate and were only slightly better than those from a simple benchmark model. Within each outcome, prediction error was strongly associated with the family being predicted and weakly associated with the technique used to generate the prediction. Overall, these results suggest practical limits to the predictability of life outcomes in some settings and illustrate the…
How strong is the evidence?
Evidence Hub reads this as a cohort study, done in people. Its evidence level is 3 of 5 (Moderate · trial / cohort / secondary analysis). Trials without a large randomized design, cohorts and secondary analyses give moderate evidence.
Read the paper
Related studies
- Characterization and Analysis of Porosity and Pore Structures
- Practical considerations for conducting ecotoxicity test methods with manufactured nanomaterials: what have we learnt so far?
- Dental Optical Coherence Tomography
- Radiation-shielding concrete: A review of materials, performance, and the impact of radiation on concrete properties
- Testing entanglement of annihilation photons
- Calanus finmarchicus seasonal cycle and diapause in relation to gene expression, physiology, and endogenous clocks
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.