If you’ve ever wondered how many pets there are in the UK or whether free universal childcare would really create 1.7 million jobs, you should talk to Georgina Sturge. She is a statistician in the House of Commons Library and it’s her job to provide MPs and their staff with the data they require. And while politicians might pepper their speeches with facts and figures that sound impossible to dispute, she knows there’s often a more nuanced story to be told.
“Numerical data is seen as more solid than other forms of evidence, yet I’d often notice that beneath the surface it really wasn’t solid at all,” she told me when we met in a busy French café a short walk from the Palace of Westminster. Her new book, Bad Data: How Governments, Politicians and the Rest of Us Get Misled by Numbers, is a whistle-stop tour of all the ways the data that forms the basis of policymaking can fall short.
Sometimes, she explained, how the data is collected leaves huge holes that make it virtually useless, such as the International Passenger Survey that was used as a guide for the number of people migrating to the UK. It was only collected at the UK’s biggest airports (Heathrow, Gatwick and Manchester), which meant it missed everyone arriving from the post-2007 EU states in eastern Europe on the new budget airlines, which ran to cheaper regional airports. The result? The 2011 census revealed that migration had been underestimated by nearly half a million people.
Or sometimes the same data can paint radically different pictures. Sturge writes about Boris Johnson and Keir Starmer clashing over whether child poverty had gone up or down since the last Labour government. The reality was both politicians were right – depending on whether you started counting from the year 2009-10 or 2010-11 (Gordon Brown left office in May 2010), there were either 100,000 fewer children in poverty by 2019, or 600,000 more.
[See also: Ha-Joon Chang: “Talk of a ‘fiscal black hole’ is an insult to the concept”]
“That’s a basic example of the decisions you make in your analysis completely changing the conclusions you draw.” Sturge fixed me with a piercing stare. “So my question would be: well, what’s the reason you’ve chosen that specific year?”
Sturge, 33, has something of the look of a tech CEO with her black polo-neck and hair pinned up high. She is not a mathematician by training, studying English at Balliol College, Oxford University, before travelling to the Netherlands to do a master’s in public policy and human development at Maastricht University. (The Netherlands is a bit of a theme during our interview – as well as providing an instructive comparison with the UK, it’s also where Sturge’s wife is from and where her wedding was held over the summer. Data, she told me laughing, was integral to their wedding planning – “I did use quite a few spreadsheets”.)
A humanities or social science background is useful for statisticians, she believes, because her job is not just about crunching the numbers but thinking about all the human factors that go into collecting them. “If you really understand where they come from, then you know a lot more about them than if you were to come at them from a purely mathematical perspective and take them at face value.”
You might want to understand, for instance, how it can be that the two main ways of measuring crime – information recorded by the police and data from the Crime Survey, which has been running since the 1980s – seem to show opposite trends. One reason is the way that successive governments have repeatedly tweaked how the police record crimes: changing targets that can incentivise officers to use their discretion over whether or not a particular incident is counted as a crime.
Meanwhile, public perceptions of what is or isn’t a crime also change over time, as do people’s willingness to report it. Comparing rates of sexual violence or hate crime to a past era when cultural values were very different (marital rape was only made a crime in 1991, while our hate speech laws are just three decades old) isn’t necessarily helpful for policymaking. So are certain types of crime rising or falling? Sometimes we just don’t know – but that won’t stop MPs from using the data to make a point.
And that’s if the data exists at all. “We naturally tend to expect that there will be data on important topics and you can find an answer on any question if you just look hard enough,” Sturge told me. “I was surprised to find that in some important areas” – inequality, benefits, unemployment, children out of school – “there just isn’t the kind of hard data you would expect.”
[See also: Richard Reeves: “We have to rescript masculinity”]
One issue is the reluctance in Britain to connect different data sources. We have no national population register – rather, data is haphazardly collected by local councils, the NHS, HMRC, the Passport Office, and a host of other organisations. The Office for National Statistics makes a sterling effort, but there are still major gaps in between censuses. Compare that to the Netherlands, where in 2011 the government joined up all the data it already had on people: “The entire ‘census’ involved 15 staff, had a budget of €1.4m and took a matter of days,” Sturge writes in Bad Data. “The UK’s 2021 census is due to cost around £1bn and will take years to process.” By which time, of course, much of the data will be out of date.
The UK, Sturge was at pains to point out, is better at data collection than a lot of other countries. Our censuses may be clunky and expensive, but at least we do them every ten years; the Democratic Republic of Congo’s last census was in 1984. And some countries don’t even have data on births and deaths – “According to Unicef, one in four children in the world ‘doesn’t exist’ because their birth was never registered,” she writes.
But we cannot allow ourselves to be complacent – because when data is missing or used in the wrong way, the consequences can be devastating. In the Windrush scandal thousands of people were harassed by the Home Office and in 83 cases wrongly deported, because the government hadn’t kept adequate records of their right to live in the UK.
[See also: Gary Lineker: “The BBC can’t stop me talking about politics”]
Sturge also cites the cautionary tale of Carmen Reinhart and Kenneth Rogoff, two Harvard economists whose 2010 model purported to show that, if a country’s debt-to-GDP ratio exceeded 90 per cent, the risk of a negative impact on long-term growth became significant. This research was used by George Osborne to justify austerity, on the grounds that if the UK didn’t get its debt under control, the economy would shrink. The only problem? Reinhart and Rogoff had made an error, missing off five rows of data on their spreadsheet, negating their conclusion about the risk of negative growth. By the time this was spotted, austerity was well under way and its effects are still being felt now. And while it’s a stretch to say that David Cameron and Osborne would have pursued a different course had the mistake been caught earlier, bad data gave them cover for their economic programme.
If we want our politicians to use data more responsibly, Georgina Sturge argues, we need to invest in better ways of collecting it. It’s perverse that we know more about the performance of Premier League footballers than how many children are out of school. She also believes MPs should be bolder about admitting when the data is uncertain or comes with serious caveats, warning that “there are so many cases where people have been confident in data that hasn’t really been up to the job”. But whether offered up by surveys, government records or artificial intelligence, the key is for all of us to keep asking questions about where the data that governs our lives comes from, she says, and what assumptions those gathering it were making. After all, “the numbers don’t count themselves. People do.”
Content from our partners
Related
This article appears in the 23 Nov 2022 issue of the New Statesman, Russian Roulette