An international research team from the Max Planck Institute for Human Development and the Toulouse School of Economics warns of the risks of using AI in the approval of social benefits. The study, published in Nature Communications, shows that affected individuals, such as the unemployed or those in need, distrust automated decisions because they can reinforce biases and lead to harsh consequences in case of errors. To gain acceptance, the perspectives of vulnerable groups must be considered in the development of such systems.
The case of the AI program Smart Check in Amsterdam illustrates the problems: The system checked social benefit applications for fraud by analyzing data on addresses, income, and family to calculate risk scores. High scores forwarded applications to case workers, but disproportionately affected migrants, women, or parents. Criticism from associations and lawyers led to the suspension of the program; an evaluation confirmed deficiencies such as a lack of transparency and opportunities for appeal.
In three surveys with over 3,200 participants from the USA and the United Kingdom, the researchers examined attitudes towards AI in social benefits. Participants chose between slow human decisions and faster AI options with a higher error rate in rejections. While many citizens accept minor inaccuracies for the sake of speed, benefit recipients reject AI more strongly – even with a 5 to 30 percent risk of incorrect decisions.
Non-recipients systematically overestimate the trust of affected individuals in AI, even with financial incentives for realistic assessments. Vulnerable groups understand the majority opinion better than vice versa. Demographic factors such as age, gender, and income were considered; the US sample was representative, the UK sample balanced between Universal Credit recipients and others.
A hypothetical right to appeal only slightly increases trust and does not change the rejection by recipients. Acceptance depends on trust in institutions: the stronger the resistance to AI, the lower the trust in government. In the United Kingdom, many prefer human processing, even with equal efficiency.
The study calls for participatory AI development that involves affected individuals, rather than prioritizing efficiency alone. Otherwise, there is a risk of incorrect decisions and declining trust in administration and technology. An ongoing collaboration with Statistics Denmark is now collecting data on Danish vulnerable groups.
At a glance:
- Surveys: Over 3,200 participants on AI in social benefits in the USA and UK.
- Differences: Recipients more skeptical than non-recipients; the latter overestimate trust despite incentives.
- Measures: Right of objection has limited effect.
- Recommendation: Participatory processes for AI systems to avoid loss of trust.
Original Paper:
Heterogeneous preferences and asymmetric insights for AI use among welfare claimants and non-claimants | Nature Communications
The comments are closed.