I heard or read some good advice re that sort of thing recently. If you walk into someone else's frame taking photos then there is a sense of invasion of privacy and cause for conflict, but if they walk into your frame then 'psychologically' there's not much they're entitled too. It seems to ring for me. I've only had a couple of times where people gave me some grief. One was obviously deranged and paranoid, and I wasn't even shooting him. I told him to f##k off and he did. The other was a grumpy old guy who walked away looking confused after I reminded him it was a public area we were both in a public area :)