No, you missed the point of the referendum. But you have understood the action proposed to be taken, which is half the story.
Essential to the referendim is the previous experience of centralising Aboriginal organisations along the same vein as the Voice was supposed to be. The problem was, from government to government, there was a lack of ongoing commitment, the organisations were subject to the political winds of the day.
The half of the story you've missed:
Putting a requirement for 'a Voice' in the Constitution sets a minimum standard that all governments in the future would have to meet. In whatever form suits them and the political tides of the day, but they couldn't shut the body down entirely without ensuring replacement of that body with another.
The government could, or may, re-establish an authority like the Voice now, but as before, it has no protection from being closed down for whatever reason, without replacement in the future.
It was a really smart proposal. The fear campaign from Advance, and the poor promotion of the case from the YES side, led to the simple and good proposal being lost in trashy scare campaigns and uplifting adverts of no substance. Not to mention the mostly horrible 'debates' that never actually covered the proposal itself and always wandered off into ridiculous areas the proposal didnt cover.
Some of the best speakers i found on the subject were Hannah Mcglade, Julian Leeser, Anne Twomey, but there were plenty more.