Need's Work
I beat every castle up to level 20 with just 50 archer's with 50 Fletch buildings and 50 upgrades to them... I found that they are the only ones that do considerable damage to the castle, I tried sending Infantry, cavalry, Rams, and the Cannons. They pretty much all died aside the ram getting through more than half of the castles hp with 50.
I wanted to try a different way and built 100 Barracks, with 100 upgrades, and bought 100 Infantry. About 60 died from the arrows while I was on level 12 and then the 40 that attacked the castle rapidly diminished from the water, I sent in the archer's after that.