The under-fire secretary of Victoria's CFMMEU is in the sights of federal Labor leader Anthony Albanese, who wants to expel him from the party.

Mr Setka launched a legal bid to prevent that, but on Tuesday morning Victorian Supreme Court Justice Peter Riordan ruled it was not within the court's jurisdiction to make a decision.

"The court does not interfere with internal decisions of voluntary unincorporated associations unless it is protecting or enforcing a contractual or other right recognised in law or equity," Justice Riordan ruled.

"The plaintiff has not established any such underlying right."

Not only did Mr Setka lose his case, he was also ordered to pay costs. He was not in court for the ruling.

However, the judge said if he was wrong and the court did have jurisdiction, the powers of the ALP to expel a member are subject to compliance with preconditions set out in the Victorian branch rules.

Mr Albanese's push started after reports Mr Setka criticised family violence campaigner Rosie Batty for eroding men's rights - an allegation the union leader denies.

In June, Mr Setka also pleaded guilty to charges of breaching a court order.

His wife Emma Walters outed herself as the victim of a series of abusive text messages from her husband.

The Australian reported over the weekend Mr Setka also threatened his wife with violence.

"If any of these more recent allegations or rumours were true, I would have been charged, but the police prosecution of me was completed," Mr Setka said in a statement on Monday.

The move to expel Mr Setka from Labor has sparked division within the union movement, as different leaders chose to support or condemn Mr Setka.

Victorian Premier Daniel Andrews continued to back Mr Albanese's position to rid Mr Setka, but would not be drawn further on Tuesday.

"There could be an appeal to that decision so I'm loathe to offer a commentary beyond simply saying that they are party matters," he told reporters at parliament.

"You know my position, I support Anthony Albanese's position on this matter. As for who leads the CMFEU, again I make the point very clearly that is entirely a matter for members of that union."

The ALP will wait for the full judgment before considering its next steps.

Australian Associated Press