It's just seen as bad luck by some because the word for "four" and the word for "die" is similar (both si with different tone) (I'm not even sure if this is Mandarin or Cantonese, I'm currently assuming Mandarin because all "Chinese" seems to be Mandarin).
Some places go as far as to not have 4 numbered floors apparently like some buildings in western countries don't have 13th floors but like any other given place they're not all over the top superstitious. So because some people have been slinging 4s like faeces your unlucky (haha the irony) choice of number got caught up in the insanity.
Though given people if you'd randomly used another number someone somewhere would probably have had something to say about it regardless XD
I've been waiting for an explanation like this all day.
And you're right. Tell someone a 2533 means something bad and the bad people will be all over it.
If you do the numerology thing where you have to reduce all numbers to single digit (done by adding them together til the answer is a single digit) you end up with 4 again. 4 on the brain! XD (and also what can be done at a stretch ;D)