definitely keep going with it
I can see what you mean about it not quite turning out the way you wanted. I REALLY like the concept though. A couple of suggestions.
1. try different levels and different challenges with different songs
2. be a little more forgiving on the circles on the first couple of levels so people have some time to get into the game. As it was, it took a while before you realized it was going to get good.
3. try adding some background art when you get round to it.
I can't wait to see what another few hours put into this comes up with.