Maybe the flag's strength should not be tied to the users SP (assuming this is how it works). Instead each user would have the same voting weight with their flag. This would require a community consensus for an account to be truly effected. That way one individual cannot destroy someone's reputation because of their singular view of how the blockchain should be used. In this scenario it would take a majority vote and this may also encourage everyone to use their flag without fear of retaliation by one single individual.