I think actually it's not THAT uncommon, but yeah, most people can't do this. There was a number, like "every (umpteenth) person has synesthesia", and the number wasn't that big.
I usually see colours when I listen to music, but then they turn into complete scenes and "musicvideos" so I don't know if that counts!