He moved south because there were beautiful buildings everywhere. "What I didn't realise was Tasmanians don't give a flying f--- about their buildings, on the whole, any more than they did about their natural environment. Their two greatest assets are the natural and the built environment, and both of these are in the process of destruction by a bunch of bogans."

His experience in Tasmania, he says, "was probably the unhappiest episode of my life".

"I think I came very close to either a nervous breakdown or suicide. I just started to fall apart."

We're having lunch in the Bridge Room near Circular Quay. It's a bit unnerving for me because Schofield is a pioneer restaurant reviewer, a celebrated foodie, a distinguished cultural figure and noted bon vivant, whereas I am none of those things. To make it worse, Schofield used to have his own regular feature, "Lunch with Leo", in Australian Gourmet Traveller magazine, in which he dined and drank with the great and the good, and also Qantas chief executive Alan Joyce, who brought his publicist with him.

Schofield, 79, has also been a successful advertising man, and he has run the Sydney Festival, the Melbourne Festival, and the Hobart Baroque festival, which this year became Brisbane Baroque. Schofield grew up in Brewarrina, NSW, a small bush town east of Bourke, where his parents owned a pub. Brewarrina used to be the end of the line of the NSW Railways network, and Schofield went back recently to refresh his memory for a memoir he plans to complete. It is even more isolated today. The line has closed and the station has burned down.

"Initially, I went to school at the local Catholic school," he says, "which was called the Convent of Mercy, run by Mercy nuns who were, in fact, merciless. Poor bitches, they were dragged out from some boondocks of Ireland and shipped out to the boondocks of Australia. God knows what sort of a life they had. They had a rough time, I think, and so did we, as a consequence. They were mad with the birch.

"When I was eight," he says, "I was despatched from Brewarrina to a Catholic boarding school which was a preparatory school for St Joseph's at Hunters Hill. Again, more nuns but a different order. They were not quite so free with the strap." His parents' pub went bust, and the family came to Sydney when Schofield was 12, and ran a "ham-and-beef shop" in the inner west "because the German word 'delicatessen' was not attractive" in the 1940s, he says.

He went to Christian Brothers' High School Lewisham, where the Brothers were "mostly reasonable", he says, "but there was one particular sexual molester who, in today's climate, would, I suspect, be immolated Savanarola-style in Martin Place for his extensive depredations. A terrible man. But we didn't talk about it very much. There was a code of silence." Schofield wanted to write, so he left school to take up a cadetship with the Herald, while he studied for a BA at Sydney University. He also enjoyed six months' national service in the air force.

"I think it should be absolutely compulsory for all young people," he says. "It would do them the power of good. Discipline is the greatest thing for health. You rose, bathed, shat, ate, worked, in that strict routine." I point out it would be courting disaster to do it in a different order.

"It was like going to a health farm," says Schofield.

He ended up finishing neither the cadetship nor the degree, and became an advertising copywriter instead. For lunch, Schofield orders ash-grilled duck. He says he doesn't usually bother dining out on a dish he can cook at home, such as pasta, and he never makes duck. Since I can't really cook anything, my choice is less restricted, but I plump for the Wagyu sirloin because I know what it is.

Schofield returned to Sydney University as the director of a theatre company, and met his wife, Anne, on one of his productions. They moved to London in the 1960s, because that's what everyone else was doing, then returned to Australia after their first child was born. They missed London's dynamic cultural life, but the arrival of twin daughters slowed them down a little anyway.

In the 1970s, while still working in advertising, Schofield began a parallel career as a newspaper columnist and restaurant reviewer. He says food criticism, like drama or music criticism, is important because it may end up as the only evidence that the performance - or the meal - ever existed.

"When I think of all the restaurants that I've known and eaten at in Sydney," he says, "they're now just memories: Primo's, Romano's, Prince's - dozens of restaurants that used to have a reputation - are long gone. And the only record that exists of the type of customer they had, and the sort of food that they served, and the quality, and where they fitted in the big scheme of Australia's development from a country that didn't care much about food to one that's almost obsessed about it now, is there in reviews.

"Dishes were incredibly popular that would not grace a menu anywhere anymore. When I grew up, the posh dishes were carpetbag steak or chicken in a basket or prawn cocktail. None of them are to be despised, they're all terrific, but they're out of fashion."

Unfortunately, Schofield's most famous restaurant review, published in this newspaper in 1984, led to a successful defamation action by the Blue Angel restaurant, which objected to Schofield's dismissal of its lobster dish as, among other things, "close to culinary crime". The restaurant and its owner were awarded $100,000 damages plus interest when the truth and comment defences failed.

"I'd like to be remembered for things other than that," says Schofield. Did the experience change the way he reviewed restaurants? "It just moderated the language a bit," he says.

But he received a lot of support, and Governor-General Bill Hayden invited him to stay at Government House, and he concedes he might have been "getting a little too cocky".

"So I think it was a good thing, in way," he says. "In the same way as I don't think what happened in Tasmania was a good thing." We'll get to that.

Schofield and his wife broke up after 19 years. "I'm gay," he says "but not in a proselytising way and many gay men would envy me for the fact that I've been able to have children." Schofield loved living in Sydney and, in time, became a minor symbol of the city, like the Queen Victoria statue but with glasses on.

"The tourism people would ring me and say, 'There's someone coming from overseas, will you tell them about Sydney?' I should've written a f------ book about the place." At the same time, however, he had long been "absolutely obsessed, tremendously intrigued" by Tasmania. He had "a romantic view about architecture" and Tasmania has some of Australia's loveliest historic buildings. "I must've made at least 60 trips down there," he says, "and I looked at houses, and I had this romantic idea of buying something down there, and eventually, a couple of years ago, I did."

Schofield, by now the veteran director of 11 arts festivals, suggested Hobart might take advantage of its grand heritage and stage a Baroque music festival. He won support from the tourism authority, and the 2014 festival garnered five Helpmann nominations, and one Helpmann winner. "They'd never had that for a Tasmanian event," says Schofield, "except for a puppet theatre."

Schofield hoped to expand the festival but the new Liberal government of Tasmania instead cut its funding by 25 per cent. "We were deeply wounded and shocked," he says. "I worked nearly two years for nothing, and never even cashed a chit for a petty-cash cup of coffee. And it was supported by a lot of wonderful people down there, who shared the vision that the government wouldn't. Anyhow, we threw up our hands and said, 'We're not going to do it anymore. F--- you."' An arts festival needs to secure performers years in advance, he says, and can't survive without some guarantee of its budget.

"The whole process knocked me about terribly," he says, "and I honestly started going to pieces. I was drinking, I was taking a lot of tablets, and stupidly driving when I was in no condition to drive. I was suffering from acute depression." One of his daughters was working in Tasmania at the time, and she and her sisters "decided on an intervention", plucked him out of Melbourne and spirited him back to Sydney.

At the same time, he was approached by Brisbane to take his baroque festival to Queensland. Astonishingly, within six weeks, they had moved the entire programme up the coast, in time for its launch on April 10. Schofield is bursting with praise for Queensland, but "still bitter" about Hobart. "Tasmania's such a beautiful place," he says. "It's blessed as no other area in this country is blessed, and yet they can't wait to dig it up, chop it down, sell it to the Chinese. All the young people leave, and the only ones left are the dregs, the bogans, the third-generation morons." He plans to reserve a chapter in his memoirs for Tasmania. "I'm going to call it 'Revenge of the Bogans'," he says.

Life and times

1935 Born in Sydney

1962 Marries Anne, lives in London

1963 Daughter Nell born, returns to Australia

1965 Twins Emma and Tess born

1993 Starts three-year run as artistic director of Melbourne Arts Festival

1998 Begins four-year run as director of Sydney Festival

2000 Receives Order of Australia

2002 Loses 70 per cent of his stomach to cancer

2002 Publishes The Garden at Bronte

2013 Founds the Hobart Baroque festival

2014 Founds the Brisbane Baroque festival