"Do I have the right to affect you in such an intimate way for such a trivial reason?"

Well, it all depends, IMO, on how much your opinion affects my overall reach. If one flag obliterates my post, than I fully agree with you.
But, thinking a bit outside the box, maybe flagging should also have a validation system, just to prevent what you mentioned. What do you think about that?