Personally, I think the system is fine as is. The most common scores I give are 2's and 3's. I rarely give a four or a 1 (only if it's crappy stick figures and rainbow colors, or something else deserving of that score)
I only give X's to extremely rascist or pornographic material that doesn't belong on NG anyway. ("Cow Poke" on the Daily top 5 a while back)
I've given maybe four 5's in my entire time spent on NG. (more than a year) These where to: Combat Instinct 2, Hostage Situation 2, and The Final Task.
Personally, all of the little llamas voting either 1, X, or 5's on movies need to be dragged out into an alley and beaten with a bat until they pass out. Mathematically, it makes sense..but it also screws over a lot of people who do a lot of good work (not "5" quality..but 3 or 4)
That's just my opinion. Toodles.