On the other hand, nothing is actually lost even though people don't vote. The reward pool is still of a certain size and if some people don't vote, the rewards are just distributed based on the votes given.

All the rewards are still distributed and that's a good thing - especially if the people not voting are assholes with poor taste.