Did you leave comment "most of the time" when you downvote? I think that cannot be the excuse. Because every Korean community members couldn't find your comment with your downvote.

Here is my answer to your question about irresponsible upvoter.

(1) consuming contents without upvote
(2) self-vote with delegated SP from others
(3) upvote the posting of SP whales to get upvote from them

What is your opinion about that?