Constrained for space, unable to host major international exhibitions and forced to leave much of its permanent collection in storage, the gallery was in need of an overhaul and Mr Lowy rallied the arts community to arms.

"This is not a luxury," then gallery chairman and one of the heirs to the Westfield shopping centre empire, said. "This is an absolute necessity. While there is no funding committed at the moment, I think that will come from both government and private benefaction."

Three years on, architects have been appointed and plans and designs released – but beyond that the gallery and the project's champion, director Michael Brand, have little to show publicly for their effort.

The grand plan would link the gallery to the Botanic Gardens, opening up land to the north of the existing gallery and dropping down towards the Woolloomooloo waterfront. It would bulk up the tourism potential of the area, burrowing out underground, disused oil storage tanks and doubling the gallery's exhibition space.

Critics such as former prime minister Paul Keating and architect Andrew Andersons have taken public swipes against the project, while a lack of public benefactors championing the proposal has left questions hanging over funding for the ambitious $450 million development.

Big money is required and big names are watching with interest: Gretel Packer sits on the board of trustees along with Andrew Roberts, who inherited the Multiplex construction fortune, and Geoff Ainsworth, whose family founded the pokie machine Aristocrat Gaming behemoth. Premier Mike Baird is being asked to tip in on behalf of taxpayers.

The gallery's new chairman David Gonski, so often the link between business and the arts in Sydney, has not gone out of his way to publicly spruik the proposal since rejoining the gallery, and declined to speak about the project to the Herald this week.

For his part, Mr Brand insists that the project is on track, merely in a "quiet phase" of fundraising, with the proposal awaiting government consideration of its business case and money to be freed up from Arts Infrastructure NSW.

Mr Brand says Mr Gonski is "doubly enthusiastic" about Sydney Modern as it's his second stint as chairman "so he knows exactly the issues we face and he's highly enthusiastic about the project".

But others are beginning to emerge to have their say. Prominent art collector and benefactor Sally Breen has broken ranks, stating categorically she is not prepared to pledge money towards Sydney Modern.

"Right now? No. I wouldn't, I would be far more swayed into putting money into art schools," Ms Breen told the Herald.

Ms Breen said she would reconsider if the AGNSW supported art schools in the way the National Gallery of Victoria has partnered with the Victorian College of the Arts.

She says there are other causes worthy of support and notes benefactor fatigue, particularly after a major upgrade to the rival Museum of Contemporary Art.

"I think there are more pressing things to be supporting right now in the arts rather than museum projects.

"I think there's a sense of exhaustion among benefactors. We've put our hands in the pocket for the MCA expansion, which has been fabulous, for the Venice Biennale pavilion. There's a sense of exhaustion I think on building projects."

Ms Breen's comments come after a Facebook exchange with Catriona Mordant, who forms a prominent philanthropic couple with her investment banker husband Simon, in the wake of Mr Brand citing Ms Mordant's husband in a recent story in The Australian.

"We've been giving briefings to Liz Ann [Macgregor, MCA director] and Simon Mordant letting them know what we're doing," The Australian reported Mr Brand saying. "Simon is very supportive of the gallery and is one of our benefactors."

Ms Mordant wrote on Facebook: "Very cheekily he's thrown in the Mordant name ... Supporters of the gallery, yes, but reads like supporters of the extension."

Ms Breen replied to the post: "He's ensured that we won't be supporting this. As you say ... it's very cheeky. Dumbfounded."

Simon Mordant told the Herald he has not been been asked to contribute to the Sydney Modern fundraising effort.

"We are passionate about the arts ecology in Australia and will review any proposal put to us," he said. "We haven't received any request to date."

The AGNSW has been trying to raise up to $450 million for Sydney Modern, which would double the size of the gallery. The proposal dwarfs the $53 million Mordant Wing at the MCA, which opened in 2012, increasing the size of the Art Gallery of NSW's rival by almost half.

The NSW Government allocated $10.8 million for a feasibility study into Sydney Modern in 2013 and a further $4 million last year. In this year's budget there was nothing, feeding further speculation. Mr Brand says nothing was expected in this budget.

Mr Brand told the Herald that a "a revised business case" will be submitted to the Government later this year. "That's absolutely standard procedure," he said.

A freedom of information request from the Herald to view the business plan has been refused.

A development application that was due to be lodged by the middle of this year has been pushed back to later this year.

Questions around progress on the project will come up again next week with the arrival in Sydney of the chief architects from Japan.

In June, Mr Brand surprised some when the publicly stated figure of $450 million for the project came back to $380 million, sparking questions about whether the gallery was having trouble raising money. Mr Brand told the Herald the original figure included maintenance work for the existing building and did not alter plans for the extension.

Mr Brand says fundraising is not an issue. "It's doing really well. Capital campaigns are very complicated things. It's about leverage – it's about people's timetables – so with a capital campaign you don't want to go out on day one and say we're going to start at zero.

"You want to sort of have everything lined up first and then build that success – we are getting great buy-in from our trustees and that will all be announced at the appropriate moment."

Does the AGNSW have cornerstone investors for Sydney Modern?

"We have them and we have signed agreements and they're extraordinarily generous – we're just not in a position to announce them right now."

He says space constraints at the gallery make the expansion a must. "The art world wants to see more art. Artists are very excited about the notion of being able to access more art here."

Melbourne and Brisbane have been consistently staging major exhibitions in recent years. The National Gallery of Victoria hosted Melbourne Now and the Ai Weiwei Warhol exhibition and Brisbane's QAGOMA has had the Asia Pacific Triennial, Cindy Sherman and David Lynch.

Archibald Prize-winning artist Ben Quilty, also a gallery trustee, is an outspoken supporter of the project.

For his part, Mr Keating lambasted the project late last year as a "land grab" entertainment complex "masquerading as art".

"Brand has spent most of his term in office constructing a gigantic spoof against the civic core of Sydney's most public and important open space," he said.

"In other words, the big-to-do is all about idolatry of special events and the provision of commercial venues for hire, rather than about curating or exhibiting art."

A 2011 report into Sydney's cultural facilities suggested the AGNSW could host more visitors by extending its opening hours instead of a new building. The gallery operates 10am to 5pm daily, and on Wednesdays until 10pm.

The government promised $600 million for arts infrastructure in February 2015 of which $202 million has already gone to the Sydney Opera House and $139 million to Walsh Bay.

Government sources say no announcement about Arts Infrastructure NSW will be forthcoming in the immediate future, with any announcement later this year.

Those sources say the NSW Government likes the idea of the plan – but wants to know if this is the best way to spend the money.