I'm beginning to lose count with how many people are having this problem with abusive flaggers. I'm really hoping this hasn't affected you, but by the tone of your post, you've been zapped by the trolls. :(
I wish I could find the article link to where the developers stated to the effect that people with less reputation who flag a post will not devastatingly affect your reputation score because their voting power is next to nothing. So worry not and let the idiots roll off your back. They're not worth the angst <3