The problem is...
Until you frontal cortex is actually fully developed, your ability to make a "rational", or "logical" decision is impaired. For women, that age is roughly 23, and for men, it is 25. That is why in some states, insurance rates drop at those ages...Take that sexism... Also, be glad that those are not the ages when we are considered to be adults...
That being said, if a 25 year old man has sexual relations with a 13 year old girl, he is manipulating the fact that 13 year old girl is unable make a well informed decision. He is manipulating her emotions, and taking advantage of her. And since children a significantly more mentally pliable and trusting than adults, it is easier to do. That and it is easier to coerce a minor out of fear of "getting into trouble", or "being bad".
On the flip side, if the individuals uses force, then it is purely the size difference, and lack of experience a child has at defending themselves.
The biggest problem is biology/physiology. We are all hard-wired/hard-coded into wanting to take care of someone smaller or younger then us. That is a part of love. Individuals who want to have a relationship with a minor have confused familial and unconditional love with romantic love. Probably due to their inability to make genuine human connections. Or a bad upbringing. They project their own need for love/intimacy onto someone they feel a connection with... someone they think, "they are just like me at that age", and try to "protect" them from what ever the adult perceives is the problem. And since love for men is generally a physical thing, and children are generally "cute at that age", the only way they can convey that love is though sex. They do not know any other way. Emotionally stunted. Emotionally immature. F'd up in the head.
Should men who take advantage of minor be punished... yes. Castrated... no. Death Penalty... Only if the acts performed are worthy of the death penalty for the acts performed on an adult... IE: brutally murdered.. peeled their skin.. cut out their organs while they were alive... etc
It is like the child getting bullied so bad in school, they end up becoming violent.
You cannot just treat the symptoms, you have to treat the cause... In the case of violent children, it is the bullies. They are guilty of emotional/physical abuse, etc... In the case of a pedophile, it is probably someone in their family... or someone very close to them... that has caused this condition. Then you have to (just like PTSD) work through it, and the individual actually has to want to get better. Because anyone who has been to a psychiatrist know, the only way you can get mentally better is if you actually want to. But that is a different story/tirade.
