Obviously, if my instance was sending my IP address off to other instances, I would get off that instance. However, downvotes are different situation. To my knowledge, for votes to work and be somewhat reliable, instances need to have a user attached to each vote. It would be very problematic if an instance was sending userless votes and other instances were just accepting them without issue. Nothing about the fediverse requires sending my IP off to other instances, whereas votes need to have corresponding users to be trustworthy.