Self-vote is neither good nor bad. It depends on the content and context.

For example, if I post something really value-adding and want to jump start the voting with a self-vote, there is nothing wrong with that, and indeed could be argued as a very legitimate reason to buy Steem for (non-abusive) self-promotion. (And indeed further argued that if I'm not willing to spend my valuable vote power on my own content, maybe I shouldn't even post it.)

On the other hand, if I post a bunch of empty comments or other worthless 'content' and self-vote them, then I am abusing the system

The only mechanism that allows the blockchain to distinguish between those two cases is having other people either agree with me and add their vote (or at least do nothing and let my vote stand), or downvote it and both reduce/prevent the payout and discourage my behavior since my vote has now been wasted.

Self-voting isn't the problem, but even if it were, it is only a small part of the problem. Paid votes are just as easy to abuse, already exist in very large numbers, and would not be stopped or even slowed down by stopping literal self-votes. So, no, that is not a solution.

As you correctly stated, downvotes have a high cost and as a result there are only a tiny number of downvotes that happen (and yes, as you say, many if not most of that tiny number of downvotes are from annoying jerks), and so naturally we have people abusing the system though self votes and paid votes all day long.

I disagree with your suggestion that the most effective way to address the core problem is by focusing on literal self-votes. Not only is it easy to evade, but even without any obstacle in place to evade it is already only a narrow subset of the problem, effectively a distraction.

At what point does needless downvoting stop?

Probably never. Any system which allows people to interact will always have jerks who want to annoy others. But it also should not be the case that what one or two jerks do matters much at all (or, indeed, the system is much too fragile).

So, again, like I said earlier, probably UIs should demand a lot more downvoting before paying attention to it and performing hiding, negative reputations, etc. (effectively ignoring the few jerks who can still downvote, but will not be able to meet this threshold very often if at all). And for that to happen, we need a lot more downvoting from ordinary non-jerks, where well-deserved, which in turn requires that good non-jerks not have to burn their valuable vote power in order to perform a public service.