I guess it's that delusion that [so many other people] can't possibly be wrong?

And that's pretty much the only thing wrong with "wisdom of the crowd" (which I'm all for when the crowd is actually wise not just thinking they are ;D)