There are two good reasons not to wear headphones - awareness and hearing protection.
First, the hearing. The in-ear buds are worse than the larger headphones, and they both can be easily damage middle-ear function. This threat increases with volume and distortion, and many small devices produce plenty of both. They have very cheap amplifiers which happily distort, and the MP3 files which they play are inherently low quality. Please be kind to your ears.
Perhaps more importantly is the awareness factor. The audio material may be distracting enough, but the listener has essentially given up most (if not all) of their ability to notice auditory clues and cues. I was hit by a bus a few weeks ago, but because I heard someone across the street gasp I was able to react and safely bounce off the GO bus door rather than being knocked over, possibly run over. This point is especially true at night, and doubly so when driving or riding a bicycle.
The EXCEPTION - sometimes when you don't want to talk to anyone it can be useful to have the earphones on with no sound playing through them. Its much easier to avoid unwanted conversation if you can "honestly" not hear the other person, or if you must ceremoniously remove them before asking them to repeat the question.