I can't argue with your reasoning there. But I would like to think that if it actually worked, at least a future government could hold the referendum to make it permanent and everyone would know exactly what they're getting, because they'd already seen how it was going to work. There's too much uncertainty around at the moment to convince most No people to change their mind, so I think the Yes crowd are facing an uphill battle.
One of the problems I see with the Yes arguments are that they often claim some part of the Voice will work in a certain way, so there's no need for alarm, but the problem is the No crowd can see it going wrong ten different ways and there aren't any reassurances coming from the Yes side showing how none of those problems could happen. The Yes side seems too focused on the one way things might work if everyone plays by the rules and nobody does anything bad, but it means they aren't addressing the No concerns about people taking advantage of the system, as politicians typically do. So without actually addressing those issues they're going to have a hard time changing the minds of the No people I think. It might be possible but they'd have to change tactics.
I do find it interesting and somewhat ironic how between the two of us, the same things are causing us to vote differently - specifically the permanent nature of the legislation, and that we are both hearing convincing points from people we respect.