Prime Minister Scott Morrison has condemned protesters who've set their sights on a statue of Captain Cook, saying the Black Lives Matter movement was not a "licence for people to just go nuts".
Activists in London have listed more than 60 statues around Britain to topple after a monument to 17th-century slave trader Edward Colston was thrown into Bristol Harbour on Monday.
A memorial of Cook as a boy in his childhood home of Great Ayrton in North Yorkshire also made the list.
The site says the British explorer "was a colonialist who murdered Maori people in their homeland".
But Mr Morrison said it was wrong to link Cook to the Black Lives Matter movement.
"Well, when you're talking about Captain James Cook, in his time he was one of the most enlightened persons on these issues you could imagine," Mr Morrison told 2GB on Thursday morning.
"I mean, Australia when it was founded as a settlement, as New South Wales, was on the basis that there'd be no slavery. And while slave ships continued to travel around the world, when Australia was established yes, sure, it was a pretty brutal settlement. My forefathers and foremothers were on the First and Second Fleets. It was a pretty brutal place, but there was no slavery in Australia."
However Mr Morrison was accused of having a "selective understanding of Australian history".
"The Cocos-Keeling Islands, which fall under the Lingiari electorate, began with Malay slaves and indentured labourers from South Africa, New Guinea and the Dutch East Indies," Labor senator Malarndirri McCarthy tweeted.
"Parts of North Queensland's prosperity was also built by South Sea Islanders kidnapped from their homes to work as unpaid or poorly paid labourers in the cane fields.
"In the Northern Territory, the pastoral industry was built on the backs of Aboriginal people who were not paid equal wages to their white counterparts, and whose legal battle for compensation for stolen wage is now in progress.
"The PM would do well to look into the history of the country he is trying to lead."
Labor's spokeswoman for Indigenous Australians, Linda Burney said slavery was "part of the Australian story".
"There was a literally a slave trade between Australia and Micronesia and other Pacific Islands including Fiji, the Solomons, Samoa, and in particular Vanuatu," she told Sky News.
"Of course very quickly there is also the fact that many of the young children that were part of the Stolen Generations were indentured labour out to farms and households, paid a pittance. And then many state and territory governments stole those wages. And that is slavery in any terms."
Mr Morrison also said the Black Lives Matter movement was being hijacked by political agendas.
"I think what we're seeing with some of these protests, they start on a fair point when they're raising issues about, you know, people's treatment in custody or things like that. Fair, fair issue. But now it's being taken over by other much more politically driven left wing agendas, which are seeking to take advantage of these opportunities to push their political causes," Mr Morrison said.
"I've always said we've got to be honest about our history. We've got to acknowledge the positive and the negative. But, you know, I think we've also got to respect our history as well. And this is not a licence for people to just go nuts on this stuff."
Australian National University history professor Bruce Scates said tearing down statues could be "cathartic" for groups who had been oppressed.
"First and foremost, these statues are obviously an affront to a significant part of thew community. It stands as a reminder of a past they don't wish to belong to. Modern Australians and Briton repudiate the values these memorials enshrine [and] as long as these memorials remain uncontested and unengaged with in the public space, they are endorsing the values of the past," Professor Scates said.
However Professor Scates said an alternative to removing statues was to have a frank conversation about the person or events it was commemorating.
"In a way these statues can almost become teaching platforms. For example, a lot of the community won't understand that Australia was complicit in the slave trade through Pacific labour, and a lot don't understand issues of violence on the frontier. These statues can be a way to understand the past and give it more context."
Removing the statue does not erase the past either, Professor Scates said.
"The Berlin Wall remained in our consciousness even when it was torn down. First and foremost we should be talking to communities wronged by these statements of white privilege about what they want to do. This conversation needs to be lead by Indigenous people," he said.
University of Sydney European history professor Robert Aldrich said statues should be seen as products of their own time.
"What might be preferable is a new plaque or display handle put up that contextualises them. That way they can continue to serve a pedagogical purposes in a way they couldn't do if they were simply removed," Professor Aldrich said.