Same here; I change my mind about upvotes/downvotes all the time (especially since I have the terrible habit of voting while I'm still reading the comment), but the ones I've done that for are still there. Heck, the first couple I see when I look at my profile are ones I did that for. Read half the comment, upvoted out of the corner of my eye while I was still reading, read the second half, went "Hoo boy, nevermind," and un-voted.
Related to that, upvotes/downvotes being public is a dealbreaker for me. I don't feel safe upvoting or downvoting anything here unless it's completely uncontroversial, because I don't want to open myself up to harassment by people who disagree.
I'll hang around for a bit to see if that changes, but I'm very unlikely to stick around if it doesn't.