The glossy 2002 film Minority Report is based on a 1956 novel by Philip K Dick, in which three humans with mutant powers, or precognitives – “precogs” – can foresee crimes before they happen. Spotting an opportunity, the PreCrime police department uses the precogs to virtually eliminate crime by identifying people before they break the law. Every so often, though, one of the precogs generates a “minority report” at odds with the predictions of their counterparts. This inconvenient detail is kept a secret. Revealing it would damage the credibility of the machine.
Though intended as a grimly speculative account of the problems of relying on imperfect predictions, the premise of Minority Report is increasingly being used by police in the UK as a blueprint for new technology. Last month, the Home Office pledged £5m in funding to West Midlands Police to develop a system that will identify individuals at risk of committing future crimes. Elsewhere, Durham police have developed an algorithm for use in custody decisions; Avon and Somerset Police, meanwhile, use predictive technology to map where violent crime might occur.
The West Midlands system, which police are currently developing with the help of private sector consultancy giant Accenture, will use artificial intelligence, statistics and police data to identify future criminals and victims of crime, and pinpoint those who might become an influential “hub” within a wider network of offenders involved in modern slavery. Though still in beta, the system will eventually “help the force… reduce cost across its operations”, “improve effectiveness and increase efficiency” and “reduce future demand for services”, according to Accenture’s website.
It’s unclear what will happen when the system identifies individuals as “at risk”; Superintendent Nick Dale, police lead on the West Midlands project, says the details are still being ironed out. But the pilot is a glimpse of the Home Office’s far-reaching plan to implement a National Data Analytics Solution across England and Wales, using artificial intelligence to predict which individuals are likely to commit knife and gun crime in the future.
The development of machine-learning algorithms, allied with cuts to police budgets, is propelling a version of the future long feared by privacy advocates and revered by technology companies. The question is no longer whether artificial intelligence will dramatically change policing, but how – and for whose benefit.
*****
In the age of austerity, government departments under pressure to do “more with less” have increasingly turned towards technology to solve systemic problems. Though the current prime minister has promised funding for 20,000 new officers, the overall trend is one of decline: since the Conservatives took office, the number of police officers has plummeted by 18 per cent, and 6 per cent of officers leave the service each year.
As the former police lead of the West Midlands project Iain Donnellly told the New Scientist in 2018, cuts have left forces searching for a system that can prioritise individuals already known to officers. To do this, the pilot pools data from the last decade on people’s criminal histories, the number of times they’ve been in custody, and instances when they’ve been stopped and searched – including, says Dale, occasions where police didn’t find criminal objects.
Predictive power aside, the practice of zeroing-in on known offenders can create a dangerous circle that compounds biases towards particular groups. In the UK, black people are 40 times more likely to have been stopped and searched than their white counterparts. An independent ethics report on the West Midlands project published in April pointed out that, although ethnicity had been excluded from the algorithm, other data sources can become proxies for the same characteristics – for example, stop and search records. In a follow-up report answering some of these concerns, the West Midlands Police clarified that none of the features of the model could be used to directly identify ethnicities, but conceded that “any process of risk assessment… will have the potential to lead to biases of some form”.
As Frederike Kaltheuner, a researcher from advocacy group Privacy International, says: “Predictive policing determines who you are, with implications for your future, based on data from your past.”
The mathematician Cathy O’Neil emphasises that unjust decisions can be reinforced when programmed into computer systems that are presented as objective. In her book Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy, O’Neil discusses the example of PredPol, the predictive policing software first used in Los Angeles that has since been trialled by police in Kent. PredPol advertises that data is geographic rather than demographic – meaning individualised information isn’t fed into the system. (The West Midlands Police pilot, on the other hand, uses data on specific individuals.) The problem is that, because certain geographic areas are already over-policed, data used by PredPol’s algorithm is already skewed towards areas where the most policing takes place. “Policing spawns new data – that in turn justifies more policing,” O’Neill writes.
Were policing merely about objectively detecting crime, stop and search might also be practiced on dog-walkers in Holland Park; police might force wealthy citizens up against the wall outside Harrods and frisk their trousers, or send SWAT teams to descend on bankers in Canary Wharf. But the expectation of finding crime in particular areas or among specific groups influences where police go looking for crime – and what they end up finding. Building computer-generated models from current policing patterns creates what O’Neill calls a “pernicious feedback loop” – where data justifies the biased assumptions that a model has been fed in the first place. In other words, the technology can effectively make real the world it predicts.
Hannah Couchman, an advocacy and policy officer at civil liberties group Liberty, contends that feeding police information about an individuals’ prior search records can encourage more zealous surveillance of groups that are already disproportionately policed. Though the West Midlands model aims to prevent people falling into crime rather than preemptively criminalising them, early intervention still increases the amount of contact an individual or community has with the police.
“Black people are more likely to be stopped and searched, so being subject to repeated stops is not an accurate indicator of risk,” Couchman says. “It’s more likely an indicator of deeply embedded and discriminatory policing practices. For biased data like this to prompt increased police contact is outrageous.”
*****
In the eyes of the law, everyone is innocent until proven guilty. But in the US, where police forces were early adopters of predictive technology, how you experience the criminal justice system depends, in part, on the colour of your skin. Predictive policing subtly distorts a presumption of innocence that is meant to be universal.
Andrew Ferguson, a professor of law at the University of the District of Columbia, fears UK police will make the same mistakes as forces across the Atlantic – where automated predictions about who will become a criminal reflect existing racial and class stereotypes.
Take Chicago, where a “heat list” is used to rank individuals according to their alleged propensity to become criminals. The algorithm remains a police secret, but the factors it draws upon reportedly include criminal and arrest history, and whether the target has been identified as belonging to a gang. The system scores individuals on a scale of one to 500 according to how “hot” they might be. Using the heat list, police then knock on the doors of those they consider most at risk, normally with a social worker or church pastor in tow.
In reality, Ferguson writes in his book The Rise of Big Data Policing, the heat list becomes more like a “most wanted” list, where predictions of risk are a proxy for areas that are already heavily policed, home to poorer citizens and people of colour. “There’s a lot of crime that doesn’t get arrested… individuals doing things inside their homes, behind their hedges, in places where nobody sees. But arrests tend to correlate to where you put your police,” Ferguson says over the phone from Washington D.C. “Chicago has a history of being racially discriminatory. Using arrests as part of your [data] produces outputs that are biased.”
“The problem with predictive policing is the policing part,” he adds. The aim of helping someone at risk of falling into crime is noble, but it also subtly shifts the remit of policing towards proactive crime prevention. In an ethics report submitted to the West Midlands Police by the Alan Turing Institute in July 2017, authors voiced concerns that the pilot is “moving law enforcement away from its traditional crime-related role and into wider and deeper aspects of social and public policy” – a move that requires “explanation, justification and legitimation”. Increasing an individual’s contact with the police isn’t necessarily the right remedy for complex social issues.
One of the most troubling elements of Chicago’s heat list is how an individuals’ risk score can also be influenced by data collected about their friends, relatives and acquaintances. An independent ethics committee voiced concerns in April about data from the West Midlands system “wrongly implicating people simply by association with other people known to offend”. The police have since countered that the use of such data “depends on the nature of that relationship, rather than because there is one at all… having a brother does not mean that both brothers are criminals, but it may be relevant in the short term”.
In the US, such associations have been used as evidence of criminal conspiracy, with life-changing consequences for those involved. As Ferguson puts it: “Your friend, your cousin, the guy you grew up with… are you guilty by proximity to their guilt because you’re in a video with them, or because you hang out with them?”
When automated risk scores are presented as neutral science, some people become less innocent than others simply by virtue of their friendship groups or where they grew up. Predictive policing distorts the universal presumption of innocence, and gives rise to the idea that some people are “pre-criminals”, who should be subject to increased surveillance.
“It’s not just guilt by association,” warns Frederike Kalthauser from Privacy International. “It’s almost the reversal of the presumption of innocence. You’re not innocent until proven guilty – you’re innocent until found suspicious.”
Content from our partners
Related
