The classification of major and minor musical stimuli along five dichotomous scales (happy–sad, pleasant–unpleasant, up–down, light–dark, and warm–cold colors) was investigated in two studies involving 51 children aged 4–6 years, and 168 adults. Musical stimuli were six chords and six harmonized melodies differing in mode (major, minor). Furthermore, fluid intelligence was assessed in both infants and adults. The associations between major mode and happiness and between minor mode and sadness increased from a proportion of 58% at the age of 4, to 61% at the age of 5, 72% at the age of 6, and 92% in adults. The major–up, minor–down associations were 62% in both the 5- and 6-year-olds, and increased to 84% in adults, while the major–light, minor–dark associations increased from 59% at the age of 5 to 72% in 6-year-olds and 92% in adults. Warm–cold colors were systematically associated with, respectively, major and minor stimuli in adults but not in children. Fluid intelligence was strongly related to the child’s ability to associate happy and sad faces to major and minor musical stimuli.