Humans are Reinforcement-Learners too

Reinforcement Learning from Human Feedback (RLHF) is all the rage in AI today. Now that I have two nieces I see regularly, I am reminded that humans are extremely sensitive RLHFs… and incredible amounts of resources are dedicated to deciding who gets to provide the feedback during the initial training run.

My sister moved recently. One of the movers was a trans-woman. My three-year-old niece was very confused. This person was broadcasting many markers of both maleness and femaleness in a way that hadn’t been co-presented in a single person in my niece’s previous training data. She didn’t know which signals should be weighted more and take precedence. She asked her mother for feedback. (“Mommy, is that a man or a woman?”)

My sister replied that this was a woman. She smiled and took my niece’s hand and said “Come on, let’s go ask her her name!”

This was very kind. It spared the subject’s feelings, and it was the socially acceptable solution. It is important for children to be socialized correctly into the class their parents think they will occupy.

However, it implies a possible shift in future generation’s abilities.

One of the most relevant facts about the sexes today is that sex is binary and easy to observe. In the future it may not be binary. Right now, it may no longer be easy for the new generation to observe! Humans are reinforcement learners. How many colors you see in a rainbow depends on how many you were taught there are. Perfect pitch is far more common in populations that use a tonal language.

It’s possible that sex is easy to observe because we received many hundreds of hours of reinforcement learning on the observable differences between the sexes. What if this training is never received? Or is deliberate confounded with malignant data?

I think it is most likely that children will eventually discover the differences between the sexes on their own. Similar to how most bright children eventually realize that God is a lie that parents pretend to believe for other reasons. Those who are like me will be very bitter and resentful for a number of years for having been misled and sabotaged for so long, but they will all eventually adapt to the reality they live in.

But what if that isn’t the case for seeing sex? What if it’s like language, in that there is a window in life were it is most easily picked up, and any learning outside that window is a struggle? There could be a large fraction of an entire generation that is significantly sex-blind the same way some people are face-blind. Or they learn to identify sex purely on some superficial marker that is easily changed (presence of lipstick? /shrug). I find this idea fascinating. Would this be net good for society? Would it be net good for the individuals who are sex-blind? Should we strive to eliminate all reinforcement learning on the matter of sex differences as a social policy? What are all the consequences of turning “sex is binary and easy to observe” into “sex is binary and unknowable to non-experts?”

This thought finally led me back to a realization of why the culture war is so important to so many people. The reason my nieces are being given reinforcement feedback that may make sex much harder for them to determine is because certain activists won this particular battle of what is socially acceptable. There is a great deal of power in being able to shape what your fellow citizens are capable of observing at the basic, preconscious level.