No shit no one alive "did anything"*, it's a euphemism for a part of history, it's intended to impart a general understanding of the transaction in a brief amount of words that sums up events. It's not intended to accuse modern people of litteral thieft.
It's okay insecure white man, no mob is going to come a knocking with a deed to your property. They didn't even have a system of written language, and your property didn't exist.
That said there will still be people alive today who either were involved in the forced separation of Aboriginal children from their parents (because there was a spate of that in 1960s still, as per the "bringing them home" report, about the "lost generations"), and/or whose grandparents and so on were involved in stuff like that. Samantha Armitages' family, and probably Gina Rhinehart's... That's part of the psychology of why some are paranoid on the issue.
But paranoia is by definition an irrational fear. The voice simply isn't about reparations.
As for the idea it will give some racial groups more power than others - again this isn't true because it wasn't just about race. Does nothing for big city Aboriginal people for instance.
It was SPECIFICALLY about people from very remote Aboriginal communities who barely count politically and are unlikely to have any affect or contact with the PM otherwise. People who can't just mount a protest in a capital city as most Australians could (90% of us live in Capital cities).
So it was about addressing a disadvantage creates by distance AND race/culture, caused by just how large Australia is (as well as our history, and why pockets of rural Aboriginal communities exist in the first place).
So nah, addressing the unfortune of being a small community that goes ignored isn't a function of over powering them or giving them a "racial supremacy".