I don't think allowing downvotes/flags to actually take away SP is a good option. It would over incentivize downvoting and create a way for groups to get together and literally rob anyone. It'll create bandits on the system. Maybe if there was fare distribution and you couldn't purchase SP, it would be possible. But not in the current state. With that said, there does need to be a better downvoting system, and it should be marginally incentivized. But the biggest thing that needs to be solved is the flag war nonsense. It's a dangerous game that benefits no one and scares smaller users like myself from downvoting spam.

The trial and jurors concept I find interesting. Perhaps this would be the mechanism that allows one to fight a flag war and protect one's self from unwarranted downvoting abuse.