This is a really nice idea!
I like the minimalism, but there are some problems.
Unlike Pong, this is too much about luck. In the original game you can cause the ball to follow a certain path by hitting it at a certain degree. Here it's almost completely random. The "ball" should change colour at a steady pace, getting faster as you keep playing.
Another problem is "hit detection". At easy levels there's a certain shade of grey which blocks almost every colour.
Apart from these minor inconveniences, this is a really good concept and a fun minigame!
Good work!