I have the downvote button hidden completely using the KES script because of that issue.
Reddit allowing you to hide your own up/downvotes to others is a really nice privacy feature that I wish the Fediverse in general had, but unfortunately the nature of federation doesn't make that possible unless you prevent them federating to other servers.