I agree with you that the term hate speech is often misused and that it is often used to shut down speech that one does not agree with. However, your analogy regarding words not being violent is not a valid argument in the slightest as it suggests that human beings are only affected by physical means. This is simply not the case. That premise denies the fact that humans are a social animal that have complex emotions and psychological mechanisms that effect their overall well being. The truth of the matter is that emotional and psychological abuse is a very real thing that can have serious consequences to an individuals mental health. Emotional and psychological abuse can lead to trauma responses and a variety of mental health concerns. An individual who undergoes extensive emotional and psychological threats will often develop trauma responses that affect their ability to live what we would call normal healthy lives.

As an example, if a person were to spend time each day shouting and screaming obscenities at their dog, the dog would develop a fear response towards its owner despite the fact that it has not been physically injured or that it does not understand the actual words being said. The dog may shake, cower or possibly pee on the floor when the owner approaches them, all of which would be a trauma response and indicate a mental health concern. Most (generalizing under the assumption that people are decent and kind towards animals) people having witnessed such abuse would likely either stand up for the animal or at least disagree with the behavior towards it. You may say that the dog is reacting to the emotions of the owner (anger...etc) and not the words. But hate speech in the real world is often accompanied by strong emotions and psychological acts of intimidation (shouts, imagery and symbols and even attire).

To further this point, the words themselves unaccompanied by emotion still affect individuals. A child growing up in a home with parents telling them (even if it is in a soft calming matter of fact voice) that they will never amount to anything, that they are useless and that no one will ever love them, likely will effect that child's long term psychological and emotional state and the child may develop a mental health issue in the moment or later on in life.

I certainly agree with you that people should be able to talk to each other and hear opinions that differ from their own and that open communication and dialogue is beneficial for humanity. But respectfully, the argument that words have no impact on humans (a social species that's largest and most influential evolutionary adaptation is communication) is simply not valid. The words are just words argument is a philosophical ideology that falls apart under real world application.

Humans are complex. They are not inanimate objects that are only influenced by physical means.