I'm going to disagree with you here. It all depends on the law. If you are in a public place with no reasonable expectation of privacy, there is no reason I need to ask for your permission.
That being said; if the man were to ask me not to take his photo, I would have stopped taking it.