I tend to call people out when they do that to me. It's irritating at best, and quite selfish at worst. You should try it a few times, when people say something like that in the comments just reply with "But not good enough for even a 1% upvote? I'm confused by your comment..." They will either stop doing it and leave you alone, or realize that it's pretty bad form to do that and stop it.