May this have something to do with the link between gonadotropin cells and the olfactor bulb?

There is a population of cells which arise in the olfactor bulb, and during our development migrate to the pituitary gland. Once there they're responsible for the production of gonadotropin releasing hormone (GnRH) which goes on to stimulate our sexual development.

There is actually a condition called Kallmann's syndrome where these cells fail to migrate from the olfactory bulb to the pituitary gland. In this syndrome people will either fail to start of finish puberty and often need hromone replacement surgery. But they also suffer from anosmia, or the inability to smell! It's pretty rare, but an interesting condition none the less.