Yeah, the flag button is rather ugly, a simple downvote would go a long way to solving the negative connotations associated with this issue. I feel the same way as you, as far as reforming my ideas about it, however I still would never use it against someone new. We have to be responsible when we have high enough reputations to do damage to others.