They are supposed to go into your ears and have direct skin contact. Did you know that the skin inside your ear is very sensitive? You really should be careful about the materials earbuds are made from. The other issue is the design. The ergonomic fit is often very bad, and they can be uncomfortable.
There’s also the question of sound quality. You’re buying them to listen to music, and sometimes the sound quality is so low that it can make listening to your favorite song seem like torture.