If only sex was as simple as a selection of gametes. There is a wide range of chromosomal, hormonal, genitalia, and physiological variation in human sex characteristics, and it is much more common than you think. And that is ignoring much more subtle variation and overlap between the sexes - cognitive, emotional, psychological - that are just as much a part of the natural variation of human sex as any other.

And before you come back with an argument about some rhetoric about “conditions” or what ever - all of evolution starts as a rare variation that becomes common in a certain population. Certain eye colors are nearly the same rarity.

Finally, there are plenty of animals that have individuals that do not reproduce. Examples are naked mole rats. We aren’t a eusocial species, but it isn’t to say we don’t have some very early characteristics of it.