Artificial intelligence (AI) underpins many areas of daily life, from chatbots that answer tax inquiries to algorithms that operate autonomous vehicles and provide medical diagnostics. A hybrid human-machine approach is necessary, claim researchers at the University of California, Irvine, to build smarter, more precise systems.
A new mathematical model that combines human and computational predictions and confidence scores is presented in a paper that was just published in Proceedings of the National Academy of Sciences.
“Humans and machine algorithms have complementary strengths and weaknesses. Each uses different sources of information and strategies to make predictions and decisions,” said co-author Mark Steyvers, UCI professor of cognitive sciences.
“We show through empirical demonstrations as well as theoretical analyses that humans can improve the predictions of AI even when human accuracy is somewhat below [that of] the AI and vice versa. And this accuracy is higher than combining predictions from two individuals or two AI algorithms.”
Researchers tested the framework in an image classification experiment where human volunteers and computer algorithms each accurately identified altered images of animals and commonplace objects including chairs, bottles, bicycles, and vehicles.
The machine classifier produced a continuous score, whereas the human participants assessed their level of confidence in the accuracy of each image identification as low, medium, or high. The findings revealed significant discrepancies between human and AI confidence across photos.
“In some cases, human participants were quite confident that a particular picture contained a chair, for example, while the AI algorithm was confused about the image,” said co-author Padhraic Smyth, UCI Chancellor’s Professor of computer science.
“Similarly, for other images, the AI algorithm was able to confidently provide a label for the object shown, while human participants were unsure if the distorted picture contained any recognizable object.”
The researchers’ novel Bayesian framework allowed them to incorporate predictions and confidence scores from both, and the resulting hybrid model performed better than either human or computer forecasts could have on their own.
Humans and machine algorithms have complementary strengths and weaknesses. Each uses different sources of information and strategies to make predictions and decisions. We show through empirical demonstrations as well as theoretical analyses that humans can improve the predictions of AI even when human accuracy is somewhat below [that of] the AI and vice versa. And this accuracy is higher than combining predictions from two individuals or two AI algorithms.
Mark Steyvers
“While past research has demonstrated the benefits of combining machine predictions or combining human predictions the so-called ‘wisdom of the crowds’ this work forges a new direction in demonstrating the potential of combining human and machine predictions, pointing to new and improved approaches to human-AI collaboration,” Smyth said.
The Irvine Initiative in AI, Law, and Society helped to promote this interdisciplinary endeavor. According to the researchers, the fusion of computer science, which creates technologies, and cognitive sciences, which study how people think and act, will give researchers more understanding of how people and machines can work together to create artificially intelligent systems that are more precise.
Additional co-authors include Heliodoro Tejada, a UCI graduate student in cognitive sciences, and Gavin Kerrigan, a UCI Ph.D. student in computer science.
Funding for this study was provided by the National Science Foundation under award numbers 1927245 and 1900644 and the HPI Research Center in Machine Learning and Data Science at UCI.
