You know, instance admins can find out who is downvoting and upvoting by checking the database. It doesn't have to be a mystery if you stand up your own instance. You don't even have to use it primarily, just get it federating your comments.