Scott Morrison has told people calling for the removal of statues of Captain James Cook to pull their heads in and "get a grip".
But the prime minister has drawn criticism for claiming Australia does not have a history of slavery.
International Black Lives Matter protests have led to the removal of many statues linked to slavery.
Asked whether he supported the removal of statues of Captain Cook, Mr Morrison said: "Cook was no slave trader.
"He was one of the most enlightened persons on these issues you could imagine," he told 3AW radio on Thursday.
"Australia when it was founded as a settlement, as NSW, was on the basis that there'd be no slavery.
"It was a pretty brutal place, but there was no slavery in Australia."
However, Australia does have a history of forced labour and stolen wages of Aboriginal people, which lasted until the 1970s.
Indigenous Labor senator Malarndirri McCarthy said the prime minister's comments demonstrated a very selective understanding of Australia's history.
Senator McCarthy said the Cocos-Keeling Islands began with slavery and indentured labour, while parts of north Queensland's prosperity was built by South Sea Islanders kidnapped from their homes to work in cane fields.
"In the Northern Territory, the pastoral industry was built on the backs of Aboriginal people who were not paid equal wages to their white counterparts," she said.
"The PM would do well to look into the history of the country he is trying to lead.
"These are all part of our nation's history, as well as the arrival of Captain Cook and the First Fleet. Truth-telling must be an integral part of unifying our country, not dividing it."
Labor's indigenous affairs spokeswoman Linda Burney said Mr Morrison's comments highlighted the importance of truth-telling.
"The prime minister's comments demonstrate a need for a greater understanding and awareness of our nation's history," she told AAP.
"We cannot achieve meaningful progress on matters such as reconciliation if, as a nation, we are not aware of the historical context of the challenges we face in the present.
"One of the crucial elements of the Uluru Statement was a national process of truth-telling."
The prime minister said Australian protesters raised fair issues about indigenous incarceration rates and deaths in custody, but said the movement was being hijacked by radical left-wingers to push other causes.
"This is not a licence for people to just go nuts on this stuff," Mr Morrison told 2GB radio.
Australian Associated Press