Domestic animals are different from wild ones in certain consistent ways. Domestic animals are more like babies -- neoteny, this is called -- with more rounded heads and so on. This is pretty much the same way that we differ from our Paleolithic ancestors; in fact a recent highly mathematical study showed that the differences between dog skulls and wolf skulls exactly parallel the differences between modern humans and those of 100,000 years ago.
The famous Belyaev fox experiment showed that you can produce all the various aspects of domestication by breeding for a single trait, the willingness of animals to be approached and touched by humans.
So the theory of self-domestication would be this: humans changed because as our population grew we had to interact more often with each other, and those individuals who were best able to interact with a wide variety of other humans had a selective advantage. Those who could not tolerate living in a group, or were too prone to violent outbursts, were cast out of the group, or killed, or just had fewer offspring. We bred ourselves in just the way we bred dogs or pigs, and the result is humans who are more like babies than our ancestors.
Or more like women. A new study by Robert Cieri et al., just published in the Current Anthropology, argues that the differences between ancient and modern humans represents the feminization of our species:
The past 200,000 years of human cultural evolution have witnessed the persistent establishment of behaviors involving innovation, planning depth, and abstract and symbolic thought, or what has been called “behavioral modernity.” Demographic models based on increased human population density from the late Pleistocene onward have been increasingly invoked to understand the emergence of behavioral modernity. However, high levels of social tolerance, as seen among living humans, are a necessary prerequisite to life at higher population densities and to the kinds of cooperative cultural behaviors essential to these demographic models. Here we provide data on craniofacial feminization (reduction in average brow ridge projection and shortening of the upper facial skeleton) in Homo sapiens from the Middle Pleistocene to recent times. We argue that temporal changes in human craniofacial morphology reflect reductions in average androgen reactivity (lower levels of adult circulating testosterone or reduced androgen receptor densities), which in turn reflect the evolution of enhanced social tolerance since the Middle Pleistocene.Their measurements show that the reduction in hyper-masculine facial characteristics took place gradually over the past 100,000 years. Their explanation is that this was linked to the decreasing importance of testosterone, either because we produced less or became less sensitive to it. (Both characteristics vary a great deal among contemporary humans.) So as we have become more civilized, we have become more feminized.
It cannot be emphasized enough that the remarkable thing about modern humans is the culture we build up by communicating with each other. Alone, our intelligence is useful but hardly revolutionary. We dominate the planet because we share information with each other; language is our salient achievement. The larger the group we can share information with, either in our own time or stretching back in time, the more we can achieve. The loner, the hermit, the wild man -- these are throwbacks to the Middle Pleistocene, dead ends leading nowhere.
I might also note that while different racial groups today vary in how domesticated we seem -- the most domesticated are certain East Asians, the least certain sub-Saharan Africans, with most European groups in the middle -- these differences are small compared to the difference between us and our distant ancestors. All modern humans are feminized domesticates.