I feel that what makes music impact people is dynamics, and the ability of the composer to share his state of mind (how he felt when he wrote the piece). It is a method of communication that doesn't necessarily require words. But, you are right that each person perceives the message differently. However, dynamics and articulation relay one common theme. For example, if I played each note of the 1st movement of the Moonlight Sonata loud and staccato rather than soft and legato at the exact same tempo, would it have the same mysterious (not necessarily sad) sound? Probably not. Music is impactful at the will of both the composer and the performer. Don't get me wrong, how the audience perceives it is also crucial, but I think that everyone gets similar feelings, they just interpret them differently. Here is a scene from the movie Immortal Beloved (About Beethoven) that kind of relays my point.