For sure. It's not fair.
There's an interesting podcast episode about a city that has a sort-of segregated area dedicated to the homeless. It's far from an ideal situation, but weirdly, has fostered a sense of community and some semblance of homes. Then again, it's still segregation of the most blatant sort. The city is considering rezoning. It's ... homelessness sucks.
https://99percentinvisible.org/episode/the-containment-plan/
Yeah, it would seem the major problem is much the same as in many other areas of life. The majority would rather not look, and rather not know.