Canberra: Daniel Andrews has criticised the prime minister, Malcolm Turnbull, for rubbishing Victoria over the so-called African gang crisis. The Victorian premier also invited the home affairs minister, Peter Dutton, to have dinner at a restaurant in Melbourne, after Dutton claimed Victorians were scared to go out.

Andrews, who was criticised by the Victorian opposition leader, Matthew Guy, for remaining on leave while reports of a gang “crisis” were dominating the media cycle, returned to work on Thursday.

He said Turnbull had not raised concerns with him about the perceived increase in crime committed by “street gangs”, despite voicing those concerns at a Sydney press conference last week.

“I might be dobbing myself in for a phone call later today, he might feel the need to now raise these matters with me, having felt the need to rubbish Victoria police, rubbish Melbourne, rubbish Victoria,” Andrews told reporters in Melbourne. “He hasn’t raised these matters with me, I must have spoken to him four or five times before the end of the year.”

Andrews said he took his family out for dinner “a number of times” during his break and did not agree with Dutton’s comments.

“I think Mr Dutton’s comments were designed to get a rise out of people ... they were designed to be as controversial as possible,” he said. “I don’t know how often he spends time in Melbourne, he’s always welcome to come and have dinner at many of the fine restaurants we have across our state.”

He would not comment on whether he believed Victoria was in the grips of an African gang crisis, saying he’d “leave it to others to run a political message”.

“I think that we’ve seen some very nasty incidents in recent times, but both as a premier, as a father, as someone who proudly lives in Melbourne’s suburbs, I am completely confident that Graham Ashton and Victoria police, with record resources, are turning this around.”

Dutton has again defended his comments and repeated his criticism of the Victorian judiciary, saying there was a “problem with some of the judges and magistrates that Daniel Andrews has appointed”.

In an interview on Adelaide radio station FIVEaa, reported in Fairfax Media, Dutton said it was “complete nonsense” to deny that gang-related violence committed by African Australians existed in Victoria.

Earlier, the Victoria police chief commissioner, Graham Ashton, who returned from six-months sick leave this week, said that any suggestion Melbourne was not a safe city was “complete and utter garbage”.

“I think everyone in this room would go out for dinner, I don’t think anyone is sitting at home with the sheets over their heads,” he told reporters on Wednesday.

Asked if media coverage contributed to the perception that Victoria was unsafe, Ashton said he would not comment on media coverage but “I think people have been concerned by what they have read and seen”.

He said police were investigating reports of racial vilification toward African Australian people that occurred in response to public commentary on gang violence.

“It is concerning to hear about racial vilification and indeed ... examples of death threats being made against law-abiding members of our community and law-abiding members of our African Australian community that are the vast majority of the African Australian community,” he said. “That is something that we want to stop as well.”

Ashton was speaking at a press conference to announce a new joint police-community taskforce, following a meeting with Sudanese community leaders. The taskforce will hold its first official meeting on Friday.

Kot Monoah, one of the Sudanese-Australians at that meeting, said the youth crime which had dominated the media cycle was “not in any way, shape, or form, related to gangs” and that negative media reports had affected young African Australians, who were being racially profiled at shopping centres and were concerned it would affect their ability to find employment.

— Guardian News & Media Ltd