Fussey made two important discoveries during these trips, which he laid out in a 2019 study. First, the facial recognition system was woefully inaccurate. Of the 42 computer-generated matches that occurred during the six deployments he participated in, only eight, or 19%, were found to be correct.
Second, and more troubling, is that most of the time the police assumed that the facial recognition system was probably correct. “I remember people saying, ‘If we’re not sure, we should just assume there’s a match,'” he says. Fussey called the phenomenon “deference to the algorithm”.
This deference is a problem, and it is not unique to the police.
In education, ProctorU sells software that monitors students taking exams on their personal computers and uses machine learning algorithms to look for signs of cheating, such as suspicious gestures, reading notes or detection of another face in the room. The Alabama-based company recently conducted a survey of how colleges use its AI software. He found that only 11% of test sessions marked by his AI as suspicious were rechecked by the school or testing authority.
And this despite the fact that these software could sometimes be erroneous, according to the company. For example, it might inadvertently flag a student as suspicious if they rub their eyes or there is an unusual sound in the background, such as a dog barking. In February, a teenager taking a remote exam was wrongly accused of cheating by a competing vendor because she looked down to think during her exam, according to a report by The New York Times.
Meanwhile, in recruiting, nearly every Fortune 500 company uses resume screening software to analyze the flood of applicants they receive each day. But a recent Harvard Business School study found that millions of qualified job seekers were rejected at the first stage of the process because they didn’t meet the criteria set by the software.
What unites these examples is the fallibility of artificial intelligence. Such systems have ingenious mechanisms – usually a neural network loosely modeled on how the human brain works – but they also make mistakes, which often only reveal themselves in the hands of customers.
Companies selling AI systems have been known to tout accuracy rates in the 90s, not to mention that those numbers come from labs, not nature. Last year, for example, a study in Nature examining dozens of AI models that claimed to detect Covid-19 in scans could not be used in hospitals due to flaws in their methodology and models.
The answer is not to stop using AI systems, but rather to hire more humans with special expertise to keep them. In other words, shift some of the over-reliance we have placed on AI on humans and redirect our focus to a hybrid between humans and automation. (In consulting lingo, this is sometimes referred to as “augmented intelligence”.)
Some companies are already hiring more domain experts – those who are comfortable with the software and who also have expertise in the industry the software is making decisions for. In the event that the police use facial recognition systems, these experts should, ideally, be people who can recognize faces, also called super recognizers, and they should probably be present alongside the police in their vans.
To its credit, Alabama-based ProctorU has made a spectacular pivot to human babysitters. After conducting its internal analysis, the company said it would stop selling AI-only products and only offer monitored services, which rely on around 1,300 contractors to double-check the software’s decisions.
“We still believe in technology,” ProctorU founder Jarrod Morgan told me, “but having the human completely removed from the process was never our intention. produced, we took quite drastic measures.
Businesses that use AI need to remember its likely mistakes. People need to hear, “Listen, this machine is unlikely to be wrong. It sure does,” said Dudley Nevill-Spencer, a British entrepreneur whose marketing agency Live & Breathe sells access to an AI system to study consumers.
Nevill-Spencer said in a recent Twitter Spaces chat with me that he has 10 people on staff as subject matter experts, most of whom are trained to play a hybrid role between coaching a AI system and understanding of the industry in which it is used. the only way to understand if the machine is actually effective or not,” he said.
Generally speaking, we can’t knock people’s deference to algorithms. There has been untold hype around the transformative qualities of AI. But the risk of believing it too much is that over time it becomes more difficult to untangle our addiction. It’s good when the stakes are low and the software is generally accurate, like when I outsource my road navigation to Google Maps. Not good for unproven AI in high-stakes circumstances like policing, catching cheats, and hiring.
Competent humans need to be in the know, otherwise the machines will continue to make mistakes and we will pay the price.
More from Bloomberg Opinion:
• Everyone wants to work for boring big companies again: Conor Sen
• Plastic recycling works, so ignore the cynics: Adam Minter
• Twitter needs to tackle a bigger problem than bots: Tim Culpan
This column does not necessarily reflect the opinion of the Editorial Board or of Bloomberg LP and its owners.
Parmy Olson is a Bloomberg Opinion columnist covering technology. A former journalist for the Wall Street Journal and Forbes, she is the author of “We Are Anonymous”.
More stories like this are available at bloomberg.com/opinion