The main problem is the shrines sometimes don't come on time. You can pay money at literally every shrine and yet still lose because you ran out of time at some point. That scenario should never happen if this was properly balanced.
The problem tends to happen early game rather than later, and tends to happen if you encounter more of those golem-type enemies early on (rather than the easier dog/wolf-type enemies). But let's say you get two or three golems in a row between very early-game shrines? Yeah you're dead because out of time.
That's probably worse than the overpowered dragon, which yes can be encountered early, but I feel is more "fair" in the sense it's a dragon and is probably set to be low percent chance any enemy could be a dragon, vs. what feels unfair because losing because running out of time due to zero mistakes made whatsoever.
If I'm to lose due to zero mistakes, it better be from an enemy defeating me, not because running out of time.