After the longest trial for an individual charged with murder in NSW history, Justice Elizabeth Fullerton formally discharged the 12 men and women of the jury on Tuesday after they indicated for the third time that they could not reach agreement.

Within seconds, the state's senior Crown Prosecutor, Mark Tedeschi, QC, sprang to his feet to apply for a retrial at "the earliest opportunity" and asked on behalf of the NSW Director of Public Prosecutions that the accused, Robert Xie, "be remanded in custody".

But Mr Xie's barrister, Graham Turnbull, SC, said his client would be applying for bail at the first opportunity and should be released.

"A single day more in custody is not appropriate," he said, adding that his client "looks forward to a retrial".

Mr Xie, 51, has been in jail since May 2011 when he was charged with the murder of his brother-in-law Min "Norman" Lin, Min's wife Lily, their two sons Henry, 11, and Terry, 9, and Lily's sister Irene.

All five victims were found beaten and bludgeoned to death with a hammer-like object in the bedrooms of their North Epping home.

Since his arrest, Mr Xie has faced trial on three occasions – the first two trials having been aborted due to the emergence of new evidence and the ill-health of the then presiding judge, Justice Peter Johnson.

He has also made several bail applications in the Supreme and local courts, one of which was successful but overturned on appeal.

If Mr Xie is denied bail again it is likely that by the time his fourth trial reaches its conclusion in late 2016 he will have spent five-and-a-half years in custody without having been proved guilty of an offence.

"My husband is innocent and we will never, ever give up," Mr Xie's wife declared as she walked from the court on Tuesday.

The comments were in stark contrast to the torrent of tears that had flowed down Mrs Lin's cheeks 90 minutes earlier when she realised that her husband would not be acquitted.

For a few moments the couple stood less than two metres apart, separated by a corrective services officer who stopped them from touching.

Mr Xie, who had until that point remained fairly composed, started to break down as he was taken away from his wife.

Mrs Lin has steadfastly maintained her husband's innocence, giving evidence in the trial he had been in bed beside her throughout the night in question.

But the prosecution said that Mrs Lin would "say or do anything to protect her husband", including telling "abject lies".

It alleged that Mr Xie, driven by a combination of bitter resentment over what he saw as the Lins' favoured status within the broader family group, and a desire to get his hands on his brother-in-law's financial assets, crept into the Lins' house using the key he and his wife had been given, and then beat the Lins to death.

Mr Xie's defence maintained that one man, acting alone and in the dark, could not have killed five people in their bedrooms without any one escaping or raising the alarm.

The jury were evidently split between the two positions, or simply unable to decide.

Their discharge came after they had written two separate notes indicating that they were unable to reach a verdict.

The second note was followed by a direction from Justice Fullerton that she would accept a majority verdict from 11 of the 12 jurors, but they were still unable to agree.

After discharging the jury, Justice Fullerton indicated that she would make an order that they be excused from any further jury service for the rest of their lives.