I think it would be possible. The software would just have to record a downvote, saying “we checked out this account and registered one downvote, and everything was valid”. The downvote is only reversible on the original instance the logged in user is on, anyway, and that’s between the user and server. The identity doesn’t have to be displayed to others on the original instance or federated.