One thing I noticed from a very brief trial of vote for vote is that the users who I exchanged votes with had no voting power left. What a complete waste of time.

I think it's great to trim down the numbers if there's nothing good coming from them. Otherwise you're left with a ton of garbage in your feed blocking out the good stuff!