I have been growing the voters list to bring in valid voters. When battles are near decision time I generally send messages out to people who have left good impartial decisions in the past and request that they judge the battle. If I see unfair voting I will bring more voters in to balance it. I look through every single vote and only count legitimate votes. If someone is being a trollish bum I probably won't count the vote. I'm also trying to get people outside of our comp to vote, and will bug people like MadFLeX, Self, and Cynic to vote as they are both solid battle rappers and not involved with the current comp.
I don't have a limited number of judges though, because when I do they often don't vote on everything and I have to chase them around for a review that might never come in. The attention the battles are getting right now is fantastic though, I mean one track can get 7 pages of reviews in a day or two. Large amounts of votes will water down biased votes.
Eventually I would like a system where votes have a weight, determined by various metrics like # of battles successfully judged in the past. Your vote may weigh less if you were voting on a friend's battle, etc. I have to program that system though, so it's not happening in this comp.
I count the votes of knocked out MCs because despite a few sour grapes the decisions are generally well thought out and impartial. Look at last year's battles, I was really impressed with how knocked out MCs judged the final rounds.
Yesh... In conclusion, I take the fairness of battle decisions very seriously. If I question the validity of votes on a battle I'm going to bring a few people I respect in to vote on them.