AI Generalization and Human Society

Fascinating piece from Scott Siskind:

In 2025, Owain Evans et al published a paper on “emergent misalignment”. They trained a previously-aligned AI to do one immoral thing: write insecure code full of vulnerabilities and backdoors. To their surprise, the AI became immoral in general. Its advice to a bored user was to try taking random expired medications and see what happened. Its money-making tips all involved theft and violence. When asked for its favorite historic figure, it chose Hitler.

Some co-authors followed up with additional weird discoveries. If you trained an AI to give the 19th-century names for birds (eg identify the American Pipit by its 19th-century name “Brown Titlark”), then the AI would behave like a 19th-century person in general (for example, assert that a woman’s proper place is in the home).

Siskind goes on the discuss AI alignment problems, and as I said the whole piece is fascinating.

But my thoughts go in a different direction: why do human preferences tend to form clusters that we identify as personality types, or political factions?

I would be willing to bet that all of the following are more common among ardent followers of Donald Trump than in the general population: American football fandom, driving large trucks or SUVs, watching "Yellowstone", interest in cryptocurrency, wanting to bomb Iran, not wanting to aid Ukraine, suspicion that the news fake and what really matters happens secretly among dubious people, cooking steaks on a backyard grill. Meanwhile followers of AOC are more likely to blame "capitalism" for their problems, listen to Chappell Roan, follow women's soccer, worry about monarch butterflies, support Hamas, use alternative pronouns, and do yoga.

You can, if you try hard enough, come up with semi-rational reasons for all of that, but I wave such rationalizations aside with the observations that while some of it makes sense, much of it does not.

I think a lot can be explained by that experiment with AI and old fashioned bird names: some things come to be associated in the culture by processes that are at least partly random, and once they are felt to go together, people of a certain sort will embrace the ones their friends share.