There's always going to be a portion of those who just take things to an undesirable (and sometimes dangerous) extreme, which is why it's important not to blanket your opinions of that bunch onto everyone else. I see a fanbase as a potential source for new music, art, cartoons, games, costumes and all sorts of other, delightfully weird shit to enjoy. One of the daughters of a co-worker loved watching MLP:FiM and began drawing them all the time. Then, it was characters from Undertale. Now, she's starting to draw her own stuff. She's even produced a couple of short, crude animations. Doesn't matter what I think of her work. She's proud of it. Am I to hate something that encourages her artistic talent, especially on a site like Newgrounds?
I don't focus upon the rotten bunch who inadvertently make something look bad by association. Chances are, those particularly detestable individuals would have behaved just as poorly without being a part of some fanbase. I mean, Steven Universe didn't encourage that abuse, and I see no reason to lump the supportive remainder of the fandom together with them. Also, I can't blame those who talk about something all the time, because nearly all of us do that with things that interest us the most. I know one person who loves talking constantly about her kids. Another about the Metal Gear Solid franchise. I'm sure I've annoyed more than a few people at work whenever I'd start rambling on and on about anime I'm currently watching. It's just something people do.
The only way a fanbase could truly ruin something for you would be if you focused your exposure towards the negativity a fraction of it emits. If people love something, let them express it. Don't call them cancer or toxic. That just makes you the unlikable fucks you're claiming them to be. I may not be part of any fanbase, but I won't disregard the positive influence one can have on some people.
TL;DR version ~ No, I don't let it ruin things for me, and I type too much.