Like ill put it this way; the 20 year old has only been out of high school for two years tops. The 24 yo has been out for at least 5 or 6. Its a big leap. After both parties are at least 25 its not a significant difference but theres a huge developmental milestone that happens around the ages of 18-22 because its the first time youre really given any real independence
Once someone learns how to stand on their own two feet and has an understanding of themselves and the outside world (usually completed in some way by 25-ish) they can stand up for themselves in a way a younger person really doesnt usually know how to. And thats why people find it attractive usually. Its extremely easy to manipulate