His boss feigns surprise. "It isn't your birthday? That's it, you're not going to get one next year."

When Shorten became opposition leader 14 months ago, Labor had just been bundled out of office after recording its lowest primary vote in a federal election since 1931. At least two terms in the political wilderness seemed certain. But the turnaround has been dramatic: recent polls have put Labor ahead of the ruling Liberal-National Coalition by as much as 10 per cent - a margin that would deliver an emphatic victory at the ballot box.

Pundits agree the reversal of fortune is due more to the government's unpopularity than to anything Shorten has done. In fact, from a public relations viewpoint, he has had a torrid year. In August, he revealed he had been the subject of a police investigation over an unsubstantiated rape allegation. The royal commission into trade union governance and corruption has heard accusations against him of branch-stacking and other questionable dealings. Nonetheless, the 47-year-old has a higher approval rating than Prime Minister Tony Abbott and looks likely to be Labor's next resident of The Lodge.

In the past, Shorten has been wary about the personal scrutiny that goes with public office. "Bill was always one step ahead of me, telling people not to talk to me," says journalist Jason Koutsoukis, who set out to profile him for a book on Labor's turbulent recent history. "Or if they were going to talk to me, he was saying, 'Get him to email you the questions.' Then they would email the questions to Bill, and Bill would say, 'This is what you will answer.' "

Koutsoukis, South Asia correspondent for The Age and The Sydney Morning Herald, admires Shorten - the two were close friends in their university days. But "Bill was really worried about what I might write", he says. "I knew that from multiple sources." In the end, Koutsoukis withdrew from the project for unrelated reasons, and the book, Downfall: How the Labor Party Ripped Itself Apart - was written by his intended co-author, Aaron Patrick. Before it was published last year, a lawyer acting for Shorten unsuccessfully demanded to vet the manuscript.

For this story, though, the opposition leader decides to make himself available. I spend time with him in Canberra, Melbourne and Sydney, tagging along as he travels from one appointment to the next, sitting in on tactical discussions and listening to his staffers prep him for interviews. It is the week before the G20 meeting in Brisbane and there is a trace of hysteria in the air. People are talking about armoured motorcades, renewable energy targets, free-trade deals and Prime Minister Tony Abbott's promise to shirtfront Russian President Vladimir Putin.

None of this distracts Shorten from the main game, which is winning the 2016 election. On a footpath in the Sydney CBD, he has a conversation with conservative columnist Gerard Henderson, who questions his wisdom in putting himself under the microscope. "I can't expect people to vote for me if they don't know me," Shorten says.

One thing we have long known about Bill Shorten: he is the consummate Labor machine man, a masterful backroom wheeler and dealer who, as head of the Australian Workers' Union (AWU) and leader of Labor's Right faction in Victoria, wielded influence in the Caucus room even before he was elected to parliament. The Power Index website once named him the nation's number one "political fixer". Kathy Jackson, embattled national secretary of the Health Services Union, has described him as "probably the most senior factional warlord in the ALP.

To meet, Shorten is genial and quietly spoken. A bit less than average height. A little soft around the middle. One of those faces that remain boyish into middle-age. It is hard to imagine him in warlord mode, but a factional ally, Earl Setches, federal secretary of the Plumbing Trades Employees Union, says it is a mistake to get on his wrong side. "When the door closes and he fires up, he's very, very tough," Setches says. "He can stare down any opponent. He doesn't like to show that side of him often, but be under no illusion. His eyes will burn through you when he starts screaming. If you double-cross him, you'll know it."

Setches adds that Shorten approaches life as if it were a chess game. His mind is not on his next move, but the one after that. Or the one after that. "He's very astute in thinking ahead to outcomes in two or three years' time, rather than in three weeks' time."

The minutiae of daily life can escape him though. One morning, he sits in the front passenger seat of his white Commonwealth car as it idles outside his electorate office in the north-west Melbourne suburb of Moonee Ponds, wondering where he has put his tie. A staffer dashes into the office and soon reappears with it. "Crisis averted," says Shorten's senior press secretary, Ryan Liddell, who is sitting in the back with a briefing sheet on his lap.

Our destination is a solar panels factory, which Shorten will inspect before holding a media conference. En route, he draws my attention to places of interest ("that's Essendon Airport on your right") while Liddell runs through questions the reporters are likely to ask, and suggests suitable responses. "You could say ... The line we're going to need here is ... You should use ridicule here ..." And so on. "If you get pushed on whether you're disappointed that the PM didn't shirtfront Putin," Liddell says, "you should say it goes to show you just can't trust what Tony Abbott says. Treat it as a whole sort of trust narrative."

Shorten lives in his electorate of Maribyrnong with his wife, Chloe, their four-year-old daughter, Clementine, and Chloe's two children from her previous marriage, Rupert, 13, and Georgette, 12. He says he misses the kids terribly when he's away from home. Fatherhood has changed his world view: having spent most of his adult life consumed by work, he now accepts that "I've got to reserve some emotional space for things other than my job. I understand, and maybe it's a belated maturity in my mid-40s, that if you don't leave space for the people who really care for you regardless of politics, you're perhaps missing something fundamental."

One of Shorten's long-time friends takes his newfound devotion to domesticity with a grain of salt. Shorten recently appeared on Kitchen Cabinet, the ABC-TV program in which a different politician each week prepares a meal for journalist Annabel Crabb. He was seen cooking ratatouille. "Pretending to cook ratatouille," says Yossi Berger, a former AWU colleague. "Which he hasn't got a bloody clue how to do. This is a charade. I mean, this is a rubber ratatouille or something." To Berger, Shorten's entire performance felt fake. "He puts his arm around Chloe, he kisses her on the cheek. This is all role-playing. You don't see the real person."

Another friend, AustralianSuper chief executive Ian Silk, agrees there is a disconnect between Shorten's on-camera persona and his true self. The real Shorten is "extremely charismatic", Silk says. "He's got a big smile, quite an extroverted personality. A quick wit. His natural warmth and charm are not communicated at the moment through the television."

Silk can say that again. Shorten's delivery of lines can be so stiff and clunky that the latest series of the spoof TV current affairs show,Shaun Micallef's Mad as Hell, included a regular segment featuring his lamest pronouncements. Shorten's staffers are loyal and protective - I watch Liddell adjust his boss's tie before a media conference, see communications director Kimberley Gardiner pat down his hair before he walks on stage to give a speech - but no one enjoyed Zingers, as the segment was called, more than they did. One day when I'm in the office and Shorten is out at a lunch, somebody replays a couple of the clips, including one in which he muddles the order of his words: "Just when I thought nothing else could this government do surprise me!" Fond laughter.

In June 2010, Shorten was one of a handful of faction leaders who organised the overthrow of Labor's then prime minister, Kevin Rudd, replacing him with the deputy prime minister, Julia Gillard. The coup had catastrophic consequences - not just for Rudd but for Gillard, who in turn was ousted before last year's federal election, and for the Labor government, which was fatally destabilised. But as he talks about it in his Parliament House office one afternoon, Shorten gives the impression that what he regrets as much as anything is the tarnishing of his image.

The night before Rudd was deposed, Shorten and a few other Gillard backers gathered at a Vietnamese restaurant in the Canberra suburb of Kingston. "Someone said, 'We'll go out and have a bite to eat and get away from the House,' " he says. "That was a mistake." A journalist who arrived on a tip-off reported that Shorten had a list of names in front of him. An ABC crew filmed him talking intently into a phone in front of the restaurant, apparently marshalling numbers for the move against the prime minister. The perception formed that he was the chief architect of Rudd's demise, and consequently much of the public resentment about it was directed at him.

But appearances can be deceptive, he says. The battery in one of his two phones had gone flat, so he wrote down the names of people who had left him messages. And when he stepped outside, it was to phone a woman named Kim Simmons, his stepdaughter Georgette's primary school teacher. No, really. "I'll put in a call to Kim," he says. "I'll see if she is willing to confirm to you that I was doing a parent-teacher interview."

Sure enough, Simmons attests that she and Shorten spoke for 10 minutes about Georgette's academic progress: "It wasn't just, 'How is she going?' He had specific questions about the comments I'd written in the report." Simmons was impressed that Shorten had interrupted his meal to have the conversation - and even more impressed when she later discovered what else was happening in the national capital that evening. Shorten obviously had a lot on his plate, besides spring rolls.

When it became clear that, under Gillard, Labor would not win the 2013 election, Shorten helped dispatch her, too. "I formed the view that we were going to be smashed beyond belief," he says, explaining his support for the reinstatement of Rudd. "And a weak Labor Party is not good for a strong Australian democracy." One of his mentors, Bill Kelty, former secretary of the Australian Council of Trade Unions (ACTU), says this was a brave decision, taken despite the risk that Shorten would draw more flak: "There was absolutely nothing in it for him. It was only going to hurt his longer-term leadership potential."

As it turned out, all was forgiven and forgotten after the election loss. Not only was Shorten anointed opposition leader, he became the beneficiary of rules introduced by Rudd to make the position virtually unassailable. Things have worked out well for Shorten. As they almost always do.

In the back of the car, Kimberley Gardiner is checking emails and monitoring breaking news. Suddenly she looks up: "There's been a huge development." Word has just come through that the US and China have struck a landmark climate-change deal, setting ambitious new targets for the reduction of carbon emissions. This is vindication for Shorten, who has backed away from negotiations with the Abbott government to relax Australia's target of 20 per cent renewable energy by 2020. "We've got this also-ran prime minister who didn't want to discuss climate change at the G20," he says. "We've got these pygmies in charge! These hopeless B-graders."

Later in the day, Ryan Liddell reads him a statement issued by the federal environment minister, Greg Hunt, in response to the US-Chinese agreement. " 'What's important for Australia,' " Liddell reads, " 'is that we have replaced Labor's ineffective and costly carbon tax with a policy that will actually deliver significant emissions reductions.' " Shorten sighs and says: "God, they tell f...ing lies."

Legend has it that Shorten has beenworking assiduously towards becoming PM since he was a teenager. "I don't remember now," he says, when I ask how long he has wanted the job. But yes, he was passionately interested in politics, even as a boy. "I'm sure I wanted to do something good. And big."

Shorten joined the Labor Party at 17, in 1985. "He was inspired to do something for the battler," says Peter Cowling, who signed up at the same branch: Murrumbeena, in south-east Melbourne. "He referred a lot to his father, who worked on the wharves." The popular notion that Shorten had humble beginnings has eased his path through the Labor movement. "Very working-class roots," says the plumbing union's Earl Setches.

But Bill Shorten snr was actually a marine engineer who became a dry-dock manager. Shorten and his only sibling, a non-identical twin brother, Robert, who now heads Orion Corporate Advisory Services, went to Xavier College, one of Melbourne's most expensive Catholic private schools. According to Victorian state Labor MP Luke Donnellan, a fellow pupil, Shorten was a brilliant debater with "an amazing belief in his own capacities. You always knew he was headed somewhere." Donnellan believes Shorten's confidence came from his mother, Ann, who died this year. "She was a very determined woman, who went back to study after being a teacher and got a law degree. She was very much the inspiration for the boys."

At Monash University, where Shorten enrolled in an arts-law course, he decided he wanted a spot on the students' association political action committee. The ALP Club put him ninth on its ticket, but he campaigned so successfully that he was elected with the most votes. "I think he approached every first-year girl who walked into the union building," says Michael Borowick, then a fellow student and now ACTU assistant secretary. In Downfall, Aaron Patrick writes that Shorten "seemed to walk onto campus a fully formed political operative. He was charming, scheming, worldly. He made wry comments about international affairs. Women liked him. He was adept at befriending younger followers who would come to idolise him."

Patrick, who was active in Network, the Shorten-led Right faction of Young Labor, and is now deputy editor of The Australian Financial Review, was one of those who fell under Shorten's spell. "I loved the guy," he says. "He was fantastic. But then, when I was no longer of use politically, he ended the friendship and we didn't have any contact after that." A former federal Labor MP says that fits with a pattern: "Bill has a capacity to burn people relatively quickly after they've delivered what Bill wants." But his friend Peter Cowling argues in Shorten's defence that while he has a gift for cultivating relationships, he simply hasn't time to maintain them all. "Bill's in a hurry," Cowling says. "He wants to do stuff. He wants to go further. That was always going to involve stepping on quite a few feet."

While still at uni, Shorten landed a job as an adviser to a Victorian Labor state government minister, Neil Pope, who remembers being bowled over by his energy and self-possession: "I was saying to friends, 'Mark this guy's name down because I think one day he might be PM of Australia.' " After graduation, Shorten became an articled clerk at Maurice Blackburn, a legal firm with close ties to the Labor Party. Managing partner John Cain, the son and grandson of Victorian Labor premiers, says Shorten threw himself into the work, "but you only had to spend a little bit of time with him to realise that he had bigger things in mind". His ambition was palpable. "You felt the heat coming from him," Cain says.

In what looked to some like an odd career move, Shorten then became a trainee organiser at the AWU, the nation's biggest blue-collar union. He tells of being sent to a scrapyard to try to recruit members: "I go into this portable hut - you know, there's the microwave, the crap International Roast coffee, those thin-skinned plastic cups which burn your fingers. There are half a dozen blokes sitting there, and one of them has a tattoo around his eye. I thought, 'You must be tough because that would have hurt so much I would've been screaming like a baby.' " The men had no interest in the union and told him to leave. "I went in and closed the door behind me and said, 'Listen, I can't go until you join.' "

Along with the scrapyard workers, he signed up racehorse strappers, agricultural labourers, even netballers. He says he found the job immensely satisfying. "Organising workers meant they got paid better, or they were safer." As he rose through the ranks to become secretary, Shorten rebuilt the union, which for years had lost members and struggled financially. A former AWU colleague says: "He was unhappiest when he was in the office doing admin. He wanted to be out connecting with the membership. He loved that."

His friend Yossi Berger recalls seeing Shorten give an electrifying speech from the back of a truck outside a Melbourne whitegoods factory that had been marked for closure. "It was directly from the gut," Berger says. "This was a very emotional and very angry situation. A huge number of workers were losing their jobs. They were in pain but they totally shut up and they listened. You could have heard a pin drop." Berger pauses. "I'm still waiting for that quality to come out in parliament."

In a submission to the ongoing trade union royal commission, Shorten said he had no recollection of an alleged conversation in which he warned a colleague not to blow the whistle on an AWU slush fund. The royal commission was also told that Shorten oversaw an employment deal that allowed a company making unusual payments to the AWU to lower the wages of mushroom pickers. And the Health Services Union's Kathy Jackson accused him of using union money to pay for Labor Party branch-stacking. He has denied the claims.

While Shorten was establishing a reputation as a champion of the proletariat, he was married to Deborah Beale, the daughter of a wealthy Melbourne investor and former federal Liberal MP, Julian Beale. They had become engaged two weeks after meeting in 1999, when both were studying for an MBA. Shorten developed a strong bond with one of the Beale family's closest friends, billionaire businessman Richard Pratt. He tells me that when he and Deborah were holidaying in the US with the Beales and Pratts, he suggested a side-trip to communist Cuba: "I didn't seriously expect that they'd say yes, but they did." The party travelled there in Pratt's private jet and heard president Fidel Castro give a speech. "It was amazing," he says.

In 2001, when Shorten spoke at the National Press Club for the first time, Pratt flew to Canberra to be in the audience. According to Aaron Patrick, "He was like a proud father."

Shorten became estranged from his own father, who had divorced his mother in 1988 and remarried two years later. Shorten says he and Bill snr just grew apart, but his father's widow, Helen, tells me that Bill snr never understood why his son had cut off contact with him. Fortunately, Shorten paid his father a visit a few weeks before his sudden fatal heart attack in 2000, but Helen says she hasn't seen him since the funeral. "He and I were supposed to be co-executors of his father's will, but he just didn't respond any time I tried to call him," she says. "I had a few personal things of his father's that I thought might have been of interest to him in the event that he eventually had some children, but I'm still holding them here."

Five days after the collapse of the Beaconsfield mine in northern Tasmania on Anzac Day, 2006, news broke that two missing miners, Brant Webb and Todd Russell, were still alive. Shorten borrowed Richard Pratt's plane ("I said, 'I've got to get down there' ") and flew into the eye of a media storm. With Webb and Russell trapped a kilometre below the surface, their union leader became the face of the drama. Shorten's frequent briefings - calm, informative, articulate - were compulsory viewing as Australians held their breath and awaited the men's rescue. By the time the two finally emerged into the light on May 9, Shorten was a national celebrity. "BILL FOR PM" screamed Sydney's Daily Telegraph.

He entered parliament at the 2007 federal election, after winning preselection for the safe Labor seat of Maribyrnong in controversial circumstances. The sitting Labor MP, Bob Sercombe, had pulled out of the contest on grounds that it was a stitch-up. "Bill will win simply because of the deals that have been done, and the locals will be ignored," he said.

After Sercombe withdrew, a senior Victorian Labor figure, Eric Dearricott, wrote in an internal party document that the votes Shorten had in his pocket included a block controlled by George Seitz, a Labor member of Victoria's legislative assembly - and a notorious branch-stacker. The Age later revealed that Shorten, then Victorian ALP party president, had chaired an administrative committee meeting that exempted Seitz from a party rule that state parliamentarians retire at 65. Shorten refuses to talk to me about branch-stacking, saying that to make any comment would "just give the allegation oxygen".

Shorten expected to go straight into the federal Labor ministry, and admits he felt short-changed when instead the new prime minister, Kevin Rudd, made him parliamentary secretary for disability services. "But after about 24 hours I thought, 'Okay, there's a job here, let's find out about it.' " Under Gillard, Shorten was promoted to higher office - he became assistant treasurer and at different times held the portfolios of financial services, workplace relations and education - but it is the work he did for the disabled that has won him most plaudits. Using his union-honed advocacy skills, he won bipartisan support for the National Disability Insurance Scheme, a plan which, when fully implemented, will see all Australians share the cost of caring for our most vulnerable citizens.

The plumbing union's Earl Setches says Shorten always had a strong streak of kindness and empathy. Setches remembers walking to a meeting with him and noticing he was carrying four copies of The Big Issue, the magazine sold on the streets by homeless people. Pressed, Shorten confessed that he couldn't pass a vendor without purchasing one. "He said, 'I've got to walk back the other way now because otherwise I'll end up buying them again.' "

Just after Shorten became opposition leader, a woman told Victorian police he had sexually assaulted her at a Young Labor camp in 1986. After the state's Office of Public Prosecutions advised there was no reasonable prospect of conviction, police said in August they would not proceed with charges. Shorten held a media conference to say there was "absolutely no basis" to the claim. "It was hard on the family," he says now, "especially when your kids read it on the internet. That's what prompted me to deal with it." [The woman's lawyer, Peter Faris, says she intends to pursue the case.]

Earl Setches, who walked the Kokoda Track with Shorten, has no doubt he has the staying power to make it to The Lodge. "He's not the most athletic man in the world," Setches says. But what Shorten lacks in co-ordination and muscle tone, he makes up for in grit: last year, he stunned the organisers of Melbourne's annual 15-kilometre Run for the Kids fundraiser by recording a time 10 minutes faster than that of the super-toned triathlete, Tony Abbott. "Phar Lap's heart beating in the body of an aardvark," Setches says.