Liking their music doesn't mean you'll automatically like the fans, or want to associate yourself with them. You know that one guy down the block who doesn't shower and whose breath could kill an old lady with a weak heart?
Sure he has some awesome music blasting out of his creepy-as-fuck house all day, but do you want to hang out with him? Fuck no. Fuck that guy. Nobody likes that guy.
*cough*