At the age of three, he suffered an acute abdominal disorder which, although not life-threatening, left him with a mild speech impediment and some neurological damage.
At elementary school, Yamashita was the victim of bullying and on one occasion wounded a classmate with a knife. Because of this, his parents decided to move him to the Yahata institution for the mentally handicapped in Ichikawa, Chiba. His IQ was measured at 68. It was here he started to experiment using torn pieces of paper to create pictures. His talent was recognized by mental health expert Ryuzaburo Shikiba, who organized an exhibition of Yamashita's work in Osaka which received wide praise.
Tiring of life at the institution, and in order to avoid the mandatory physical examination for recruitment into the Imperial Japanese Army, Yamashita ran away in 1940 to start his wandering around Japan, which would last until 1954.
At the age of 21, staff from the institution found him helping in a restaurant and forced him to take the recruitment exam. Eventually he was considered exempt from service. The events from this time were recorded in his “Wandering Diary” of 1956, and the most popular image of Yamashita travelling alone through the country with his rucksack comes from this period.