Are we taking Woody Allen for granted?

In some ways, Allen is a prisoner of the independence from Hollywood he fought for so long to protect.

Do you know what a state Annie Hall was in when it first emerged from the editing room? Maybe you’ve heard that its original title was Anhedonia – referring to Alvy Singer’s inability to experience pleasure – but it wasn’t just a title. That was the film that Allen shot: a Fellini-esque stream of consciousness, honeycombed with flashbacks to Alvy’s Coney Island childhood, featuring a murder mystery, a Nazi interrogation dream, an elevator trip to hell and a basketball game between a team of philosophers and the New York Knicks.

“Terrible, completely unsalvageable,” said Allen’s co-writer, Marshall Brickman, of the film they saw as a rough cut in late 1976. Only one thing worked: the subplot involving Alvy’s romance with Annie Hall. “I didn’t sit down with Marshall Brickman and say, ‘We’re going to write a picture about a relationship,’” Allen later said. “I mean, the whole concept of the picture changed as we were cutting it.”

His reaction to the success of Annie Hall – his biggest hit at the box office at the time and the winner of four Academy Awards – was the same reaction he had to any of his films that went over too well with the public: he disparaged it, while quietly absorbing its lessons. Bits and pieces of Annie Hall showed up in his other films for the next two decades – Alvy’s Coney Island childhood resurfacing in Radio Days, the murder mystery in Manhattan Murder Mystery, the elevator trip to hell in Deconstructing Harry – while reshoots and rewrites became a staple of most of his pictures, granting him the freedom almost of a novelist working through successive drafts.

“It was remarkable what he did for me,” Diane Keaton later said of Allen’s ear for Annie’s Chippewa Falls language: self-conscious, neurotic, a little jejune in her attempts to sound smarter than she is, “flumping around, trying to find a sentence”. Annie Hall was a breech delivery, as indeed it had to be, as the first film of Allen’s that was almost entirely taken over by another performer, a voice other than his. As a kid growing up in Brooklyn, Allen studied the great magicians and in many ways his greatest achievement as a director has been to make himself disappear.

Introverts often grow up thinking that they are invisible – a fear, perhaps, but a strangely comforting one and something of a sustaining fantasy should they become famous. These days, Allen has the invisibility of ubiquity, noiselessly producing a film every year for critics to take a whack at: is it good Woody or bad Woody?

Allen is a figure occluded by the scandals and speculation of his private life, which still sends tabloid Geiger counters crackling, some two decades after his break with Mia Farrow. The headlines could almost be the pitch for a Woody Allen film, were it not that Allen has already made it. In Zelig, the chameleonic hero is, you may remember, “sued for bigamy, adultery, automobile accidents, plagiarism, household damages, negligence, property damages and performing unnecessary dental extractions”, before finding redemption in some Lindberghian derring-do – an accurate forecast, in a sense, of Allen’s return to making crowd-pleasers in the mid-1990s. Except that Zelig was released in 1983. On the rise and fall of Woody Allen, Allen, it seems, was there first.

His 46th film opens in cinemas on 11 September. In Irrational Man, Joaquin Phoenix plays Abe Lucas, a dishevelled, alcoholic philosophy professor who decides to pull himself out of his funk with a spot of murder, which has long replaced masturbation as the favoured activity of the Allen male. I’ll leave it to Allen’s old shrinks to tease out the connection between comedy and murder, spotted by Freud in Jokes and Their Relation to the Unconscious – why else do we talk of comedians “killing” it, or “slaying” their audience, if not for the release of hostility common to both? And I’ll leave it to the critics to decide the relation of Irrational Man to the earlier Match Point and Crimes and Misdemeanors.

The problem with late Allen is not that the films are bad necessarily but that they are sketchy: spindly and dashed off, the result of a too-easy passage from page to screen. Allen’s has to be the shortest in show business. A film a year, as regular as clockwork, with zero studio interference. He is the one genuine success story to emerge from the big, hairy, super-freak auteurist experiment of the 1970s – the auteur of auteurs. Francis Ford Coppola crashed and burned. Martin Scorsese crashed and came back. Robert Altman was driven into exile, Terrence Malick into early retirement. Who would have guessed that the only film-maker to keep chugging along would be the writer of What’s New Pussycat?

It may tell us something about auteurism as an idea, certainly as a production model in Hollywood, which has always reacted to success by throwing money at it, granting film-makers ever greater control – a dubious drug denying them the artistic constraints and collaboration in which their creativity first flourished. It vacuum-packs their talent.

The one-man-band aspects of Allen’s career mask the juice that he gets from his co-conspirators: Keaton, but also Dianne Wiest, Farrow and Judy Davis. Most of his biggest box-office successes have been co-written: Annie Hall and Manhattan (with Brickman), Bullets Over Broadway (with Douglas McGrath). “The first thing he says is, ‘If you’re not comfortable, change it,’” said Wiest of working on Hannah and Her Sisters.

“It’s as if he’s got a feather in his hand and he blows it and it goes off in a dozen directions,” said Jeff Daniels after starring in The Purple Rose of Cairo. It’s a lovely image, for that is what the film is about: the unruliness of creation running disobediently beyond its creators’ grasp. This is the great Allen theme. It is the theme of Bullets Over Broadway; of his other great farce about artistic creation, “The Kugelmass Episode”, his New Yorker short story about a professor of humanities who drops into the pages of Madame Bovary to conduct an affair with its heroine; and of his one-act play Writer’s Block, in which the characters of an unfinished manuscript push open the drawer and take over the author’s Connecticut house. It is the theme of all of the romances, too, in which women grow, Pygmalionishly, beneath the green fingers of the Allen male, only to outgrow and leave him.

The biggest dead patches in his work, on the other hand, have come when he was most cut off from collaborators: the run of movies he made in the late 1980s and early 1990s with Farrow, clenched in silent agony and overdosed in brown; or the series of comedies that he dug out of his drawer for DreamWorks in the early 2000s – The Curse of the Jade Scorpion, Hollywood Ending, Anything Else – long after he had lost interest, or could summon the energy for farce.

In some ways, Allen today is a prisoner of the independence from Hollywood he fought for so long to protect. He encourages his actors to change his scripts as much as they want, but who is going to pluck up the courage to tell the quadruple Oscar winner that kids don’t “make love” any more, or fall for “nihilistic pessimism”, or name-drop O’Neill, Sartre and Tennessee Williams? Jason Biggs, the star of American Pie and American Pie 2 and Allen’s lead in his 2003 film Anything Else? I think not.

One should, however, resist the temptation to give up on him. Midnight in Paris moved with the sluggishness of melted Camembert but Blue Jasmine had the leanness of a cracked whip, in part because in Cate Blanchett Allen found a collaborator willing to go the distance with him on a theme close to his heart: female vengeance. “Take after take after take of very exhaustive, emotional scenes,” recalled Alec Baldwin. “I sat there at the end of the day and thought, ‘She is unbelievable.’”

If Allen’s early films mined comedy from Thurber-like fantasists and romantic Machiavels and his mid-period work drew rueful comedy from reality’s refusal to co-operate, his late work seems most preoccupied by the painful urge to peel the world of illusion, to see it stripped bare. He is now at work on his 47th film, starring Blake Lively, Kristen Stewart, Jesse Eisenberg, Parker Posey and Bruce Willis – and the excitement there is surely at the thought of Willis, once the king of the wisecrack and exploding fireball, now 60, collaborating with a film-maker deep into his own twilight. Both men could well find each other’s groove, or, better still, shake one another out of it. Yipikaye, pussycat.

Tom Shone’s “Woody Allen: a Retrospective” will be published by Thames & Hudson on 11 September