If you click on a keyframe you will select its content. Doing so will change your selected color in whatever you selected.
To work around this, you will need to make sure you go to a frame without selecting it.
Use keyboard shortcuts "<" and ">" to go a frame forward of backward. Or instead of selecting the keyframe click the timeline (where the frame numbers are). This makes sure you won't select any keyframe and you keep the color you selecting that you want to use to fill between strokes.
Hope that helps!