AS THE 1996 election campaign began, Paul Keating needed to record a five-minute statement for a free advertising slot on ABC TV. It all went well enough, but the script fell thirty seconds short of the agency’s requirement. What to add? With the PM’s principal adviser, Don Russell, I drafted a paragraph concerning Medicare, childcare and the new maternity allowance – domestic stuff.

The Prime Minister was having none of it, especially the maternity allowance: he wanted to talk, as he would throughout the campaign, about Australia’s future in Asia. The debate that followed went along these lines:

Most Australians are getting on with their domestic necessities and don’t give a hoot about Asia, we said.

Well they should give a hoot. And 90 per cent of them won’t have a baby this year, he said.

But the ones who will have babies also have husbands and parents and in-laws, and half of the Australian people are women and half of them think you’re so interested in Asia you don’t give a fig about them. They reckon you’re out of touch, we said.

I’m in touch with the idea of two billion people getting richer on our doorstep, the greatest opportunity the country will ever have, he said. That’s what I give a fig about.

We were right of course – on the day. Australians didn’t want to hear about an Asian future. Their being ‘in Asia’ they neither welcomed nor believed.

While we were right, Keating was more right. Take away a maternity allowance. Feel the difference? Now take away China.

That is the difference between Paul Keating and most other political leaders: while the others believe political life depends on being in touch with the people – or shall we say ‘working families’ – and will daily go to almost any length to prove that they live and breathe the fears and hopes of ordinary voters, Keating lived and breathed the future. He had a personal relationship with it. While his opponents were holding early morning doorstops, he was conversing with what might be. He would never have settled for ‘Moving Forward’: ‘Wagons, Ho!’, perhaps, or ‘Go Forth!’ He believed this was the duty of high office, the proper ambition of any government, the fulfilment of Labor’s promise – and a lot more fun than channelling the gripes of talk-back callers.

This is not to say that Keating’s ‘big picture’ had no place for ordinary folk – including families, working or otherwise. ‘Our big picture has people in it,’ he once said, to point to the difference with John Hewson’s picture. And his picture did – it even had the arts in it, and manufacturing, and enterprise bargaining, national superannuation; an elegant little social democratic republic in the south seas, its economy freed up and powerful, integrated with the new Asian powerhouses, its society fair, tolerant, multicultural, modern and creative.

So he’d jump on the radio to talk about this future, and point out that while the rest of the OECD countries still had one foot in the mire, we had a record-breaking run of growth, and low inflation and falling interest rates, plus Asia – the opportunities were without precedent. And someone would ring from Leeton or Toowoomba and say their lollyshop was struggling to stay afloat, or tell him that women in her street were having babies just to get the government benefit and he’d say: ‘Well if you’re not making money in the present conditions probably you’re in the wrong business’ or, ‘What do people want?’ and this dark proof of his heartlessness would be playing on every news service before he got back to the office.

It’s a curious thing, isn’t it? The people cry out for vision. Where’s the vision, the media complain? And when they’re given vision, the people give the visionary about 25 per cent in the polls; and the media gives him a hiding. That’s where vision can land you. And it’s not just the media or the mob, but the nature of politics. John Hewson had his vision and politics demanded that Labor destroy it. Keating had one and even as it came to some kind of fruition he went out in a landslide.

For most politicians the ‘vision thing’ is just that – a thing you buy, hire or borrow when you need it. For Labor politicians the mantras grow ever more desperate: from ‘The Light on the Hill’ to ‘Moving Forward’must go down as one of the great political decrescendos, and very likely also one of the great dying falls. But vision was part of the indivisible Keating organism, as much a part of him as memory or teeth, and defended no less fiercely.

A dozen other stories would illustrate the point: the clash with Malaysia’s Prime Minister Mahathir for instance, or the two or three days after he addressed the House on the republic when John Howard looked lost and palpably visionless. Keating had wanted to go after Howard right then, expose his smallness. And when Mahathir was concocting a profound insult from the word ‘recalcitrant’, as if it were a synonym for ‘scheming rotter’ or ‘vainglorious bigot’, Keating had not wanted to take a backward step. Howard and Mahathir – like tariff walls or the RSL – were in the way of the future: there could be no mercy and no compromise with those who would hold us back. On both occasions his staff persuaded him to at least partially stay his hand – and on both occasions, I think now, probably they were wrong.

Of course, this is not to say that Keating was wholly averse to compromise, the occasional somersault or, for that matter, throwing a bucket of money to Tasmanians when their votes were needed. He was a politician: he dealt in power and practised the art of the possible. He was a conviction politician, which is a much rarer and generally more useful category than a conviction non-politician. Seeing things through a longer lens –and often a clearer one, it must be said – than most of his colleagues and advisers, he tended to weigh them differently, and act in ways that more orthodox practitioners found at best bemusing, at worst exasperating. Ditto, naturally, average voters. Keating inspired powerful emotions: love, loathing, rage and wonder among them – and he could inspire them in a same person on the same day.

WITH KEATING THROWN out by the people in 1996, Labor’s new shrewdies reasoned that the smartest thing was to leave him where he had landed. At the mention of his name his successors would clear their throats, shift uneasily from one buttock to another, stare into the middle distance where they imagined working families were and declare with passionate conviction that they were about getting on with the challenge of keeping the government accountable. No doubt a lot of the party operators felt more comfortable in Keating’s absence; they could get back to telling their shadow ministers what the public wanted, tutor them in anodyne lines and confected vernacular, discourage spontaneity, character and independent thought – or thought of any kind for that matter.

Having for so long been dedicated to the proposition that one governs by listening to public opinion, Labor now seems to have lost entirely the fundamental skill of changing it. At every major step they fail, at least partly because no one charged with the responsibility can paint a picture of anything that they have not seen reflected in a poll. It is not that they lack the instinct to reform, but a language and a story in which to couch it.

They would have preferred to remove Keating surgically, like tissue from the brain. Alas, such an operation was impossible. Keating was integral to the organism; not only the central character in the story, but the story-teller. Keating’s had never been the only good mind in the party leadership, but it was probably the best; he was not the only one with a vision, but his was the clearest and most articulate; he was not the only good politician but he was by far the most deadly.

They couldn’t talk about their recent past without him so they stopped talking about it, including the legacy of heroic economic reform, which just at that time happened to be bearing the fruit that Keating always said it would. Abandoning this, they also abandoned the example of leadership on which those reforms depended. In daring not to speak his name, they abandoned the achievements, ambitions and character of his Prime Ministership – the social and cultural model he was determined to weld to the new economic order. They abandoned the vision, the future. All in all it was a fair bit to give up in the name of political pragmatism. Without Keating, Labor was not entirely headless or featherless or without intestine – but it was certainly not the animal it had been.

And in all this they lost the one thing that all political strategists believe to be essential for success – a narrative. Too bad, one presumes they said; we’ll start one of our own. Nothing happened on that front for ten years or more, unless we count losing to Howard as a narrative. By the time that chapter ended, a few brave souls, encouraged by economic commentators who had even begun to say that Keating’s recession was indeed one that we had to have, and perhaps noticing that Labor’s brave legislative record had co-existed with political success, had begun to venture that they could do worse than try to reattach themselves to their own story. The problem, most obvious in the mining tax, the carbon tax and the last election, seemed to be that they didn’t know how to tell it any more. If they jump at shadows cast by focus groups it might be because until now they have never articulated the story well enough to believe in it themselves.

HOW TO WRITE the story of the storyteller? Had writer been my only role in it, I could have gone about it in the conventional way, which is to say I could have approached the personality and the events from a position sufficiently elevated to see all with Godlike clarity and authority. I have no doubt that one day such a magisterial book will be written by a good historian with time and space between herself and the subject. But the belief underlying Recollections of a Bleeding Heart (Random House, 2011) is that this way is not open to an insider – not to this one anyway. In the Afterword to the new edition I have tried to explain why. In short, the reason is that the other way didn’t work.

I tried. In the beginning I drew back from events and tried to write from the public record and planned to talk to the many people whose lives and careers had intersected with Paul Keating’s. I intended a conventional history, for which a diary I kept in the four years I worked for him would serve primarily as an aide memoire.

But after writing many thousands of words according to this method – in the third person, like a fly on the wall or God, depending on your cosmology -the whole thing was insufferably cute and unconvincing. It had a tone analogous to a policeman speaking to a camera or a court in that ghostly official phrasing: ‘The Prime Minister was observed proceeding in a homewards direction at approximately 5.39 pm.’ What I wanted was to find a vein that allowed me to capture the shades of drama – or comedy – in his going home. It was, after all, a nightly exercise among his staff to ambush the poor man before he left. A disinterested biographer writing from a conventional distance is unlikely to see this, or think it worth recording: yet it remains vivid in my mind, part of the canvas, a key to the historical moment that I was trying to describe – a minor key, perhaps, but a key.

I came to the conclusion that as far as possible I should write about what I saw – from the deck of the ship, not half way up the mast. Not Boswell to his Johnson: not amanuensis, or factotum – or magisterium. Now, you might say that without this longer view I could not discern a pattern, or give the proper weight to policy or describe the big picture of Paul Keating’s vision. That may be so: I hope not, but if I failed in this, perhaps I also managed to avoid writing a book that was entirely parti pris or littered with premature judgements. As Zhou Enlai thought two hundred years too soon to judge the consequences of the French Revolution, the five years between his retirement from politics and the writing of this book was surely too soon for definitive judgements about Paul Keating.

To put it as simply as I can, I had been privileged to see Keating and his office close-up, to be part of it, and I did not want to surrender that experience to the conventions of one narrative form. And in giving up the authority implicit in a grander or more conventional perspective, I gave up authority that, as an insider, I never had.

My job had been to draft speeches, policy statements, press releases, articles and the like, and to do this in terms that were direct, engaging and persuasive, concrete wherever possible and in the active voice. I poked my nose in many other places but writing was the main job. I seem to remember Christopher Pearson using the unlovely title ‘chief ideologist’ – deputy or assistant chief ideologist would be close to the mark.

Among the consequences of this, was the one described by a mutual friend: ‘The trouble for Paul is that the book’s about you, not him.’ Well not quite, but it’s easy to see his point. In Recollections I regularly reported what I had felt about policy, the disputes with colleagues, and the very occasional tiny victory: but my voice, drawn from the diary, was meant to play the part of chorus. In the book when I said what I wanted, what Idisapproved of or felt dispirited about, I did not intend it as objective criticism, but as running commentary on my own state of mind – a stream of consciousness, the author’s own internal talk-back. It’s true that I believe we should and could have done more, and that there were things we might have done differently: and from one week to the next every sentient member of that office – like any office, including Ricky Gervais’s Office – would have felt the same way.

Still, writing as far as possible from the angle and distance of my actual experience, I fancy I was able to write with more effect about the chaos which, properly understood, enables us to see more plainly the nature of the order that public life demands. I wanted to capture the Turneresque moment in the storm, as much as the picture of Constable-like calm on the other side of the double doors. I wanted the temper of a political office, the myriad imperatives of daily life in such a place, the way ideas form there – or don’t – and what happens to them when they are tossed into an altogether different environment; that is to say the mediaenvironment, thence to many others. Will they fizz like phosphorus in the air, flicker and go out, vanish without trace or create an unexpected storm, destroy one’s opponent or become a self-harm incident? I was hoping to make the office, and the House and Canberra itself characters in this thing. And I wanted to record as well as what was thought and done, the opportunities missed, the still-born plans, the ideas that did not form – the serendipity at work.

Above all, by writing from close up, I thought I could explore some of the tension I spoke about at the beginning: the struggle between his grand vision for the country, and the paralysis that his enemies and the realities of politics daily tried to cast upon it. This above all, I hope, is the core of the book.

And then there was the character of Paul Keating. It was around Keating that all the rhythms and vibrations gathered, and to them my memory clung – still clings in fact. Close up meant the charm, the moods, the gait, the grin, the eyes, the clothes, the anxieties, the jokes, the delicate aesthetic sensibilities, the bludgeoning of enemies. Above all, it was his language. If you want to know what Australian politics lost when it lost Keating, you may as well start with the language. He was the poet of the side, the one with the irresistible metaphors. I tried to convey this in the Afterword to Recollections:

What made Keating rare in politics then, and would make him a genuine freak now, was his ear: I don’t mean for music, but for language. He lived in the language, not as a pedant does, but in a visceral, intuitive way. Language – and music – kept alive in him the essential understanding now largely lost among our leaders that, as well as economics and politics and management and polling, there is a poetic key to human reality.

I HASTEN TO say that Labor’s current position is not something that can be fixed by employing new speechwriters. Leaders are the prime requirement. There are different kinds of leaders, but all the successful ones have nerve and self-possession and much else besides. A speechwriter may have none of these things and yet be useful, but you cannot be a leader without them. And no speechwriter, or any other kind of adviser, can make up the difference.

As Paul Keating’s speechwriter I drafted most of his speeches. This is an unexceptional fact. It would have been exceptional if, having been employed to write speeches, I later revealed that I had done most of the electrical work in the office. I wrote some of them alone, some in collaboration with other advisers and public servants, some after varying degrees of collaboration with the Prime Minister. Some of his speeches he ad-libbed, some were from notes I’d given him and to which he’d added, in some cases he read the speech and extemporised as well. Ashton Calvert, Allen Gyngell and John Edwards also wrote speeches.

To say that I or anyone else wrote them is not to ‘claim’ the speech. On the contrary, all speechwriters acknowledge that the maker of the speech owns the speech. John F Kennedy’s speechwriter, Ted Sorensen, put it this way: ‘If a man in a high office speaks words which convey his principles and policies and ideas and he’s willing to stand behind them and take whatever blame or therefore credit go with them, [the speech is] his.’ Like every other speechwriter, I hold to the same principle. Essentially, a speech is just another kind of advice, but formally composed. The leader decides how much, if any, he will take. Satisfaction for a speechwriter or an adviser is to have advice accepted: happiness is the moment when the leader makes a speech his own.

I say all this in the hope – forlorn in some quarters I expect – it might help to correct a misconception. The misconception is not that in office modern political leaders always write their own speeches – this is widely understood. The mistake is to think the audience cannot know that a speechwriter might have been involved, or that an adviser might be responsible for one or two of the ideas, and still take the speech to be an unalloyed expression of the speechmaker’s beliefs and intentions, and a measure of his intelligence, character and his sense of humour. In understanding this, the public merely see the truth of the matter.

It is on the content and form of the speech and the degree to which they were touched and engaged by it, that an audience makes its judgement. They want to see themselves and, even more than this, they want to see that the speechmaker sees them and their circumstances – recognition is a big part of it, and a big part of the speechwriter’s task. The magic occurs – as it did, for instance, with Kennedy’s Inaugural and Whitlam’s 1972 Blacktown address – when they recognise themselves in the speechmaker’s vision. At that moment, even if only then, he becomes a leader.

Ted Sorensen drafted John F Kennedy’s most famous speech, his Inaugural Address. Though it has always been widely known that Sorensen was Kennedy’s principal speechwriter, the credit for the speech, which until recent times was unreserved, has gone entirely to Kennedy. We don’t know in which man’s mind the famous line was conceived: ‘Ask not what your country can do for you, but what you can do for your country’, and really we don’t care because it belongs to JFK. While working with him on the inaugural speech, Kennedy told Sorenson to ‘drop all the domestic stuff’ and concentrate on the foreign policy. (‘[W]ho gives a shit if the minimum wage is $1.15 or $1.25, compared to something like Cuba?’ he once said to Richard Nixon.) The foreign policy emphasis Kennedy insisted upon was encapsulated in this famously resounding passage: ‘Let every nation know…that we shall pay any price, bear any burden, meet any hardship, support any friend, oppose any foe to assure the survival and success of liberty.’ The credit for these words also lies unequivocally with the man who spoke them: indeed, no one over sixty can read them without hearing Kennedy’s voice.

However, in Grand Expectations – an account of the three decades following World War II and a magisterial history if ever there was one – James T Paterson does not praise Kennedy for the words, but blames him for the wildly inflated and disingenuous hopes they express. Those words, he says, became the battle cry for a tragic folly that cost perhaps two million lives, and warped US society for decades. Bear in mind that even as he uttered them, Kennedy knew Vietnam was an unwinnable war. He told the journalist Charles Bartlett that the US didn’t ‘have a prayer of prevailing there’, but he didn’t have a prayer of being re-elected if he gave the country up to the Communists.

THIS IS A long diversion I know, but I cannot think of a better way to illustrate the point: no matter who conceives or orders them, political leaders own their words as much as they do their other decisions, because they own their purpose and owning this, they own the consequences.

For all that, they make decisions only for their time. They cannot decide how history will be written, because even if a hagiography or autobiography sells a million copies, no one, not even a president or a prime minister, can be his own judge. History will judge them – that’s their fate. It will judge their speechwriters in a footnote, if at all – that’s their good fortune.

• Don Watson delivered this speech at the National Biography Awards, State Library of New South Wales, on 8 September 2011. His book, Recollections of a Bleeding Heart, was recently reissued by Random House with a new Foreword by Carmen Lawrence and Afterword by the author.