I like keeping both. While I would agree that using the downvote is the most logical. Flagging represents a greater escalation in my own mind. One may be used for content the other for behavior.