Distressing scenes unfolded near Forrest Caves beach where a large group had been enjoying a day out on Wednesday afternoon.

Emergency services responded to reports of four people struggling in the water about 3.30pm.

Three women in their 20s and a 40-year-old man were pulled unresponsive from the water at the exposed beach known for horrendous rips.

First responders administered CPR on all four, but three of them died at the scene.

The sole survivor, a woman in her 20s, was transported by air ambulance and was fighting for her life.

She was taken to the Alfred Hospital in a critical condition, according to Ambulance Victoria.

The beach is on the ocean side of Phillip Island. Photo: 9NEWS screenshot

The Age reports that more than a dozen distraught family members and friends were huddled at the beach car park on Wednesday night as covered bodies were loaded into vans.

A middle-aged man in a white singlet was inconsolable, repeatedly yelling, “I should have stopped them” and cried out “I’m sorry, I’m sorry” while others tried to comfort him.

The family and friends of the victims were very upset once they arrived on the scene, a witness told 7News.

“There seemed to be a group of about 10 people who would have come here for just a really nice day and obviously it’s ended in tragedy,” Alex Tzatzimakis told 7News.

“(They were) crying and comforting each other.”

Phillip Island is a popular holiday destination about 90 minutes from Melbourne, famous for its penguins and the MotoGP.

The Visit Phillip Island tourism body said the caves are formed by erosion of the cliffs and are only accessible at low tides.

The waves at high tide are promoted as “for experienced surfers only”.

Police will prepare a report for the coroner and are not treating the deaths as suspicious.

– The New Daily with AAP