The new battle of ideas: How an intellectual revolution will reshape society

Why we need a radical model for society in which the state does things with, not for, its citizens.

We are living in a society which has been in manifest crisis. Prolonged crisis is both an indication that the ideas that have guided the society are flawed and a stimulus to rethink them. While our political parties have been flailing around in the minutia of chaotic emergencies, an intellectual revolution has begun. The looming battles of ideas will be as much within the parties as between them: between those who cling, and those who leap.

We should start by recognising we live in a world too complex to be fully understood. The best we can do is to learn by experimentation, copying what works – pragmatism. The concept is radical uncertainty, on which John Kay and Mervyn King have written an important eponymous book, published last year. In writing it, the authors recanted a set of established 20th-century ideas that they once taught. Integrity compelled them to leap.

The idea of radical uncertainty is complemented by a new account of how we come to understand the world, and how we form our values, by Joseph Henrich, the polymath who heads Harvard’s department of evolutionary biology. Don’t be put off by the confusing title, The Weirdest People in the World (2020). The key insight is that we have evolved to get most of our ideas not from our own sharp observation and analysis, but from the people in our socially networked communities. We have evolved this way because usually it is sensible: the collective mind of our network, which includes ideas stored for years, has observed much more than any one of its members.

Bringing these two ideas together, we rely mainly on our collective mind, but due to radical uncertainty it is seldom completely right.

One disturbing implication is that even bad ideas can be self-fulfilling, the fancy term being “ergodicity”. The ideas of a networked community generate the behaviour of its members; that behaviour aggregates into socio economic outcomes; and if those outcomes are consistent with what the ideas predicted, the whole sequence repeats.

Whatever the initial ideas, as our only partially understood world careers off like an unguided missile, outcomes will move away from expectations. This should lead to rapid revision of the ideas, but often it doesn’t. Instead, the community invents neutralising propositions that explain away the unexpected. In Iraq, the “mission” was not “accomplished”; the explanation being that “we needed more troops, so a surge”. Labour crashes to an unprecedented defeat in the 2019 general election; the explanation: “the northern working class are xenophobic and stupid”.

My own favourites: in 2018 the Turkish lira collapses as President Erdogan splurges to win the election; and Wirecard, the German payments provider, collapses as accounting fraud is exposed by British journalists. Both receive the same explanation: a shadowy network of foreign speculators. These neutralising propositions, which our partisan journalists churn out daily, can sustain bad ideas for a while, until they are finally overwhelmed by disaster.

Another quaint implication is that, in the face of the unknown, we need wisdom, a concept returning to social science after a long absence. If truth – complete knowledge of the world – is unknowable, wisdom is the best we can do: it comes from synthesising the generic knowledge of the expert with the contextual knowledge of the practitioner. Expertise is codified and readily shared, whereas context can only be understood by doing: in economics it’s “learning by doing”; in sociology “lived experience”. So, synthesis depends upon sharing expert knowledge with practitioners. Hence the practitioners should have the power to take decisions. Experts who believe they know enough to take the decisions themselves are dangerously overconfident, and so highly centralised decision structures – whether modern Whitehall or Marxist systems of central planning – are a recipe for unwise choices.

An implication important for progress is that, on rare occasions, an individual will have a new idea that is a better approximation to reality than the collective mind. Progress depends upon that disruptive new idea spreading. We now know from a valuable study of how ideas travel through networks, How Behaviour Spreads (2018), by Damon Centola, that even good new ideas only spread across communities if there are substantial overlaps between those communities. We all need to escape from our echo chambers and listen to each other.

Before you next insult some “enemy” group, ask yourself: how many people in it do you know personally? Have you tried to understand their point of view? Do they fit your stereotype? Is the typical working-class northerner really xenophobic? Are Tories really less moral than the typical Labour supporter? I like to think of myself as occupying the hard centre, and I have, in my time, been adopted or vilified by both sides, left and right. The application to politics of the loyalties appropriate for football strike me as comic. But they are also poignant: people are pouring their passions into beliefs that are too uncertain to bear the weight.

But the most uncomfortable implication is that different communities will coalesce around different ideas and values. In societies partitioned into different social networks, such communities will coexist, and depending on their ideas, they may be cooperative or violently antagonistic. Indeed, a common neutralising proposition is for each community to account for societal failure by blaming some other community. The cultivation of hatreds, though repellent, is strategic.

“Before you embark on a journey of revenge, dig two graves,” Confucius said. But that, unfortunately, is the journey on which the Labour Party’s activist membership appears to have embarked. Its composition has skewed disastrously far from the much larger group of voters the party needs to attract to win again. And the party membership is a small group, around 1 per cent of the electorate – “We are the 1 per cent!” The cultivation of hatreds against opponents is always viewed by most people as repellent. Two decades ago, with their denigration of those on benefits, it was the Conservatives who were seen by the apolitical majority as the “Nasty Party”, and it was hugely damaging.

***

Just as the Tories are overweight in bankers, Labour is overweight in lawyers, and they seem to regard the main function of the law as protecting minorities from majorities. But the law’s main function is the opposite: to enforce adherence to those common purposes that have been widely agreed by a democratic process, protecting them from free-riding by small but recalcitrant minorities – whether it’s employers that won’t pay the minimum wage, “influencers” peddling hate speech or crooks peddling drugs. Sometimes the law needs to protect people from the state, but again primarily this is about protecting the majority from state capture by well- connected insiders trying to impose their own agenda on everyone else. The law should be restraining the power of lobbies.

We do indeed need to protect individual rights against the abuse of power by the state; but often we don’t. Here, an insightful book is Hilary Cottam’s Radical Help (2018), which is already on the desk of many directors of social services. Under the guise of safety, old people are being removed from their homes against their will, and children are being removed from parents presumed guilty of child abuse unless they can prove their innocence. The children are placed in “care” (in its Orwellian usage). Such, indeed, is my lived experience: I lost a mother and later gained two children through it. My mother died to safeguard social services from liability should she have a home accident over a weekend. Bundled with my father into a care home, she fell and was dead in four days, having survived at home for nearly 50 years. Our children are our delight, but their birth-mother’s tragedy.

***

For a society to continue functioning well, it will occasionally have to change its values as better, new ideas spread across its communities. Such changes are “inflection points”; moments when the direction of the collective mind reverses. In his landmark book The Upswing (2020), Robert Putnam studies the balance between individualism and communitarianism in America over the past 140 years; swings between the “me society” and the “we society”. Putnam is a communitarian, but not extreme: both communities and individuals need rights, and periodically one overreaches at the expense of the other. He identifies two such inflection points, the first around 1900 when the Progressive movement began to reverse a descent into a rampant “me society”. Its ideas spread rapidly precisely because it was bipartisan, and based on forging new common purposes within local communities: the clingers in both parties resisted.

The process that forges common purpose is a participatory dialogue: the Irish use citizens’ assemblies, and Marvin Rees, the mayor of Bristol, holds city gatherings. In a dialogue, participants have equal voice, equal respect, an obligation to listen and to understand differing perspectives, and a purpose: to find goals that gain wide agreement and develop a plan for how to achieve them. Economists mock this aspiration; the inevitable outcome is supposedly “the tragedy of the commons”, where rampant self-interest frustrates collective action. Yet the economic Nobel Laureate Elinor Ostrom has showed that, time and again, real communities are able to avoid this “tragedy”.

The other, less happy inflection point for Putnam was the 1960s, when soixante huitards such as myself, while imagining that we were virtuously “tearing down capitalism” (which survived), turned out to be ungluing social connectedness (which didn’t). Inadvertently, we helped begin an accelerating descent into a society in which people have become even more obsessed with asserting their own superior virtue than they were in 1900.

To get a deeper understanding of virtue than the teenage antics of my generation, I recommend Virtue Politics (2019), a revolutionary reassessment of Renaissance humanism as an inflection point, which the author James Hankins explicitly compares to the Progressive movement. Central to all virtue ethics is the recognition that our minds are formed not by formal legal institutions, but by the habits of daily life.

Like other mammals, humans are inclined to be selfish, and bad habits can drag us down to being little else. But we are a highly unusual mammal, far more pro-social and creative than others. At our best, we override our individual interests to achieve common goals. Our daily experience in the organisations within which we live and work can either drag us down or lift us up. An overarching moral task in a society is to build organisational cultures pervaded by a practical ethic of decency.

The leaders of our organisations should regard their positions not as a prize to enjoy, but as a duty to fulfil. Their role is not as super-smart commanders-in-chief pulling levers and issuing orders, but as people whom we come to trust, who can then be communicators-in-chief, inspiring us to new common purposes and the mutual obligations needed to meet them. The collapse of trust in politics and business has left most politicians and CEOs unable to play that role.

***

The common message of Putnam and Hankins meshes with Henrich’s analysis of how societies can best cope with radical uncertainty. Exhibit A is Denmark, consistently among the best societies in the world on every objective and subjective measure of well-being. Agency is widely distributed around and within Danish organisations, which are continuously trying new ways of achieving their society’s common purposes. Successes spread readily between communities, unimpeded by dogmatic adherence to outdated ideas.

Sometimes the spread of successful new ideas needs leadership; sometimes it can happen by osmosis. Thanks to trusted political leadership, Danes swiftly accepted the new common purpose of protecting their neighbours from Covid, thereby missing the first two waves and rapidly bringing the third under control. In Britain, it has mainly been through osmosis that thousands of volunteers have used YouTube videos to train themselves to administer vaccines in pharmacies. Similarly, the creative invention of an immigrant entrepreneur – chicken tikka masala – spread by imitation to rapidly displace fish and chips as the nation’s top takeaway. But will the remarkable success of the Michaela Community School in north-west London – which is enabling deprived inner-city children to achieve stellar results – spread by osmosis as its unfashionable ideas for motivating children are enthusiastically imitated by other teachers? Stand by for the neutralising propositions that denigrate its achievement.

***

And so finally to Michael Sandel’s The Tyranny of Merit (2020), which has two game-changing ideas. One is to remind us why Michael Young, the architect of Labour’s 1945 manifesto, not only coined “meritocracy” in his futuristic 1958 work The Rise of the Meritocracy, but depicted it as a dystopia. For Young, if the successful came to see themselves as meriting or deserving their success, those who failed would be condemned to see their failure as equally deserved. They would be left “naked” of dignity, and so it has proved.

As shown by Paul Dolan, LSE’s brilliant East Ender psychologist, the middle classes’ narratives of success have so dominated the media (which they control) that their disdain for the supposed inadequacies of the working class – “xenophobic and stupid” – has become pervasive. The activist Labour membership, especially in London – the lawyers, journalists, teachers and academics – exude an excessive confidence in their own abilities and thus their own opinions. It is doubly exaggerated: not only is human knowledge inherently inadequate, but part of their income accrues from capturing the rents of metropolitan agglomeration – the advantages of living and working where they do – which rightly belong to others.

The other Sandel game-changer is his concept of “contributive justice”. In forging common purposes, not only has everyone in the society the right to participate, but they also have mutual obligations to contribute to those purposes. It is these obligations that generate the rights. At the heart of contributive justice is agency. We thrive when we have some purpose in our lives other than self-indulgence, and the ability to contribute to it. And so, “distributive justice” (rights without obligations), which was a sensible agenda for the mid-20th century, will not be enough for the mid-21st.

Contributive justice is the shift from doing things for people – Blair’s de haut en bas Delivery Units – to doing them with people. The former is well motivated, but intrinsically contaminated by the patronising depiction of people as victims to be saved by saints. Contributing to common purposes provides the dignity of mutuality: everyone has an active role to play. But to contribute, people must be in a position to do so, and after decades of negligence, many are not. Rectifying that neglect is the foremost practical post-Covid priority. That combination – equality of respect; equality of participation in determining common purpose; and equality of standing as a contributor in achieving it – are what Sandel means by “equality of condition”, distinguishing it from both the Blairite equality of opportunity, and the socialist equality of outcomes.

There need be no tension between contributive and distributive justice, as long as they are recognised as generating distinctive obligations, and hence different rights. The agenda for contributive justice is the process of building a dense web of obligations that are mutually accepted and reciprocal. Elinor Ostrom’s work not only showed that there need be no tragedy of the commons, but also set out the principles of communitarian governance by which it was avoided. Such governance can work at many levels – both within small communities and across entire societies – and a polity needs its citizens to be able to cooperate at whichever level is best suited to the task at hand.

If it is to reduce loneliness, local is best; if it is to enable Yorkshire to catch up with London, then cities must work together; if we need a national effort, the entire citizenry has to cooperate. Citizenship is not merely a legal concept of rights plus an obligation to obey the law. It is about willing compliance in national purposes that have been agreed through a participatory and inclusive political process. No state can be effective without it. The first of Ostrom’s principles is the need for clear and accepted boundedness of membership. Even very demanding rights and obligations can be agreed as long as they are mutual, with everyone knowing that they apply to all. Hence, the distinction between citizens and non-citizens is existential. Willing compliance with the state depends upon a sense of mutual belonging to the nation. Claims to be a “citizen of the world” at best reveal a misunderstanding, and at worst are a ruse by the successful to deny their obligations to their less fortunate fellow citizens: an opening for the pantomime of moralising celebrities.

While the essence of citizenship is reciprocity, distributive justice is inherently asymmetric: citizens can agree among themselves to accept obligations to non-citizens, by piggybacking on the political process built for common purposes. Helping the poorest societies catch up, and encouraging rich societies to help them do so, have been the guiding purposes of my own working life. But any rights they create for non-citizens can neither legitimately impose obligations on them, nor presume any mutually participatory politics. Hence, distributive justice is derivative of contributive justice.

***

So much for the new ideas; now for their application to practical policies that address the current anxieties. The temptation is to return to what Labour once was: the “cradle to grave” agenda of public service delivery. That mid-20th-century programme certainly transformed my life, from my caesarean birth in a newly NHS hospital, through my schooling at a newly state grammar school, through Oxford on a full needs-assessed grant, followed by an Economic and Social Research Council grant for my graduate work. I don’t mean to deride that agenda. But it was, and is, a top-down agenda, something done by the state to its grateful recipients. That worked for a population accustomed to wartime obedience.

But now, the demand for obligation-free rights has exploded into absurdities: Piers Corbyn, Jeremy’s brother, and the Conservative MP and former Brexit minister David Davis, are united in demanding the right of crowds to attend open-access gatherings during a pandemic. And it ignores agency: the voters Keir Starmer needs wanted to “take back control”, and not just from Brussels. That is what Labour should now offer them, because they want to be given agency over their lives.

And anxieties, not grievances, must be the focus. Anxieties are forward-looking, and have been accentuated by objective uncertainties such as the Covid crisis and Brexit; grievances are backward-looking, subjective and consequently infinite – figures as privileged as Donald Trump and Ivy League students claim victimhood. The Welsh nationalists are demanding reparations from England for the Industrial Revolution: Sheffield should compensate Swansea?

***

The Policy Institute at King’s College, London, recently released a national poll of concerns about widening inequalities and post-Covid priorities. It found there is one – and only one – new common purpose. It is not just the top priority, but the only dimension of inequality on which there is majority agreement across communities. It is the need to reverse widening spatial inequalities: where you live should not affect your life chances. The King’s College researchers described this as “unexpected”. Unexpected after the Brexit vote, the European Parliament elections shock, the Red Wall collapse?

The most evident spatial inequality is the privileged position of London versus almost everywhere else in the UK. Government is ridiculously overcentralised in the capital. In consequence, the distance between people and influence – the number of steps in the “I know someone who knows someone” chain – is enormously skewed in favour of the London middle class. It is not just government that is overcentralised: so is finance – the economy is a hub without spokes, in the telling imagery of Andy Haldane, the chief economist of the Bank of England. Two-thirds of Britain’s venture capital for small and medium-sized enterprises (SMEs) goes to London and the south-east, a ludicrous mismatch between the distribution of enterprising people and the finance to fuel their ideas. These are the rents of agglomeration to which I alluded earlier. But spatial inequalities are also intra-regional: Blackburn suffers while Manchester thrives, and similarly Wolverhampton vis-à-vis Birmingham, and Rotherham vis-à-vis Harrogate.

Avoiding such spatial divergences would once have been relatively straightforward: Britain has become a complete outlier in the OECD. But renewing places once they have been broken is much harder. They suffer from a syndrome of interdependent and reinforcing problems, such as social deprivation, outdated skills, few high- productivity jobs, the exodus of youthful talent, an inflow of those on benefits, a bad external image and demoralisation and division among their communities. This can be changed, and elsewhere in the world there are plenty of examples, such as Pittsburgh in Pennsylvania, and Newcastle in Australia. It takes a package: devolved authority; nationally provided resources; a local leadership which fuses generic and practitioner knowledge into a convincing narrative that builds common purpose; new commitments from national leaders; visible early changes on the ground; learning as you go. Such packages work by resetting local expectations from anxiety to hope, and external attitudes from dismissal to interest.

Bringing agency and spatial equity together, Labour needs unequivocally to commit to devolution: reject Old Labour’s fondness for centralisation, embrace Gordon Brown’s proposed commission, for England and Wales, as well as Scotland. It must also recognise that agency should also pass to the voluntary sector, which has been the real hero of the Covid saga – and that jobs are generated by business, which is starting to clean up its act.

***

I will close with a related inequality: reversing the prolonged de-skilling of the half of the population not destined for university, for which past Labour and Conservative governments share the blame. We should have a single lifetime entitlement to public finance for post-school training, whether it is used for going to university, a college of further education (CFE), or any accredited training course. Labour should no longer pander to the middle classes on student fees. The SNP squeezed CFE budgets in its struggle to finance free university education in Scotland for the European middle classes while charging English students for the right to be educated in Scotland.

Equality of condition would be a real Labour Party agenda for the many. Parity of respect; a policy agenda that the many have set; parity of agency in contributing to it; and parity in the ability of places and occupations to make their contribution. The clingers will cling: they will be shrill in their opposition, and quite probably they will be nasty. But if they succeed in shackling the Labour Party to ideas that are past, they will doom it to perpetual defeat and to being engulfed by a future it neither comprehends, nor to which it can relate.

Paul Collier is professor of economics and public policy at the Blavatnik School of Government, Oxford University. His most recent books are “The Future of Capitalism” (Allen Lane) and “Greed is Dead” (Allen Lane, co-authored with John Kay)