A temporary fence erected by council on Wednesday last week at the old Presentation College site in Moe has failed to deter people from entering the premises with power continuously running inside the building.
A woman who has lived near the area for three years told The Express she had seen two men and a lady entering the site on Sunday night with two cars, one without unregistered, parked on the kerb.
The Express can confirm sighting two red cars parked in front of the premises and that the building's roller doors were also open.
"It [fencing] was useless they just lifted it up with one hand," the woman said, who requested not to be named for safety reasons.
Last week, council, together with Moe police, evicted a total of 11 people living inside the "derelict" building for the third time since February.
The woman said she was concerned for her teenage daughter's safety as she walks past the area to go to school on the train.
She said her daughter often saw needles lying in the area when she walked past.
"It worries me I wish they would just bulldoze it," she said.
The woman said people in the neighbourhood regarded the former convent as the "rough corner of the area" because of antisocial behaviour coming in the area.
She is now planning to move her family to another location as she feared that the neighbourhood had started to break down.
"It will drive a lot of people out of the area" she warned.
Latrobe City Council general manager city development Phil Stone said it was the building owner's responsibility to "vacate and secure" the premises.
"A building order can be further enforced through the courts where deemed necessary," he said in a statement.
Mr Stone also said that only the property owner had authority to request that power be cut at the Hawker Street site.
"As the building is privately owned, the power cannot be cut unless it is at the request of the owner," he said.
The Express has attempted to contact the property owner for comment, without success.
In a separate statement, council chief executive Gary Van Driel said the old convent building was "unsafe and unsuitable for multiple dwelling use".
"This derelict building does not provide the amenity and safe living conditions to a standard that our community would reasonably expect in 2018," Mr Van Driel said.
St Vincent De Paul Society Gippsland central council president Ian Hardy said he worried about the welfare of the people living inside the building and called for more government funding to improve the availability of public housing in the Valley.
St Vincent's has acted as the main charity that the Hawker St tenants access for food and temporary shelter support.
"There are a lot of vulnerable [people that are] being used by landlords who know their vulnerability and take advantage of the situation," Mr Hardy said.
"I hope at some stage it will be rectified."