The ninjas and the pirates.
They make it basically impossible to do well on one or two of the levels. How exactly does a pirate dodge an explosion he was in the center of? Also, 5 seconds is far too much on the ninjas. Make the levels bigger or reduce the time.
If you want to make the game more challenging, don't make cheap enemies. You could lower the fear and freeze percentages or lower the damage on the towers. Or you could make enemies with more health or armor. The enemies that aren't pirates or ninjas have far too little health and armor to be hard. Aside from those two enemies which circumvent every single tower in the game, I can literally just sit here and build only chilling crypts.
But all things considered, it's still a fun game, I liked it.