That's right... power is the problem.. either upvoting or downvoting... It is not fair a person to decide not to have a reward at all and kill your rep or to give you a great reward...

It is fair for a whale though to have more power but not that more...

So I just thing that there should be a less but fair amount of power for whales to upvote.

There should be not downvote but flagging . Flagging should destroy reputation BUT maybe each person should have only one vote for the same person. Maybe not even a post flag.. Like "Follow"-"Mute"-"Flag"
So when you flag the other person loses some rep and also his posts are fading out for you... then other persons have to downvote him...