A lot of the games I have issues returning to is from a bad ending or any stupidly difficult parts in the game. My memory is hazy of a lot of games I played, but here's a short list of what I'm never returning to:
Mirror's Edge
When a game with such a gorgeous art style and such fluid, intuitive mechanics has a story that is as predictable and cliche as progressive trance, it's a travesty. I played the brief story with hopes that the narrative would improve somewhere along the lines, but to no avail. Would I play a sequel if one were to be released? Absolutely, because it's rare for an IP's first game to be spectacular on all fronts, and I can give Faith a second shot, but this particular title won't get that anytime soon.
Fable II
On the topic of IPs that are spectacular... Fable was a game that proved to me that Diablo 2 had competition in terms of gameplay, graphics and story, and when I finished the tumultuous story, I was yearning for more. It was years before I played Fable II and while the story and gameplay improved nicely, and the graphics were improved exponentially, it all led to one of the most disappointing endings I've ever witnessed.
You prepare for the biggest fight of the game like you would after a ridiculous amount of grinding to reach the boss fight, and as he is doing his speech before the fight like they typically do, and then one of your teammates kills him. There was no fight. There was no fuss. Only rage. After your teammate slays the boss, he says something snarky, along of the lines of "he would never shut up" and the game ends. Fuck everything about that ending.
The Outfit
The game's about a crack squad of specialized commandos during WWII that are attempting to take out a Nazi group. It's essentially just another game about WWII, but what sets it apart from others is that it incorporates 3rd-person action gaming and RTS elements. The characters were strong and the graphics were good for a game so young in the console's lifespan, but the biggest letdown was how much potential there was for the gameplay.
The way it was played was truly meant for multiplayer, and seeing as the game is so old and was essentially unknown from the get go, there was no real drive to play it past the so-so story. Shame, really... the game could have been as popular as, say, Call of Duty?