I would keep voting.

How much your upvote is worth varies based on many different factors. How much steem is worth at the time you vote, how much your steem power is worth, and how many other people are voting at the time are all factors. Every time you vote on something, it resets the calculation which sees how much votes are worth, which is why it seems that your upvote is making things go down, but that is not the reality, it is just updating the calculation of the many factors that go into voting.

If someone with high STEEMPOWER votes something, than the price of steem goes down, and then you come in with your small upvote and reset the math, it seems like if your upvote is causing a negative effect, but it is not. The math will reset before payout anyway, so you restraining from upvoting does not help.

Keep upvoting, and use a helper like steemworld.org/@randomeno to make sure you are not using too much of your steem power.

It is complicated, but as long as someone is not coming in behind you and purposely downvoting who you upvote, than your upvote is worth something.