Spotlight Leader: The danger of the Black Box

It is already the case that engineers and researchers are unable to explain the ways in which the AI algorithms they've created reach the conclusions that they do.

By Spotlight

There are many examples of fiction for previous decades that, with hindsight, seems to have predicted the modern age, but few appear as prescient – and from such a distance in time – as EM Forster’s The Machine Stops, written in 1909. The story begins with a scene that many have argued presaged internet use by a century. A woman sits alone in a room, surrounded by technology. She can summon food or entertainment at the touch of a button, but she is bothered by constant notifications from an unseen community – “she knew several thousand people” – but she never sees anyone in person. In a line every internet user will relate to, she says: “Here I am in the dark, wasting my time”.

But Forster’s story may come to seem even more prescient in the years to come. We already live in an time in which most people are not able to explain more than a superficial amount about the technology by which they are surrounded every day. This is a new situation for humanity, and a confusing one, but it is still possible to learn how things work. As artificial intelligence becomes more sophisticated and more widespread, people will begin to use technology that no human is able to explain.

In fact, this is already happening. It is already the case that engineers and researchers are unable to explain the ways in which the AI algorithms they’ve created reach the conclusions that they do. There are engineers developing self-driving cars who are not able to get the information about why, in a given instant of a journey, the car decided to act in a certain way. Deep learning techniques applied to medical science appear to be very good at predicting the onset of some diseases, but because the software has written itself, researchers are not able to say how they work.

Few technologists worry about a robot uprising. But there is increasing unease that if the technologies that run our world become inexplicable, we may stumble into the situation Forster predicted: “No one confessed the Machine was out of hand. Year by year, it was served with increased efficiency and decreased intelligence. The better a man knew his own duties upon it, the less he understood the duties of his neighbour, and in all the world there is not one who understood the monster as a whole.”

To access the full Spotlight supplement, click here.