Actually, our downvotes don't federate out, and we don't get incoming downvotes either, so you could totally make them private within an instance since that information doesn't leave the instance anyway.
Not advocating one way or the other, just pointing out that it technically could be an option.