Previous studies have shown an association between cognitive decline and white matter integrity in aging. This led to the formulation of a “disconnection hypothesis” in the aging-brain, which states that the disruption in cortical network communication may explain the cognitive decline during aging. Although some longitudinal studies have already investigated the changes occurring in white matter microstructure, most focused on specific white matter tracts. Our study aims to characterize the longitudinal whole-brain signatures of white matter microstructural change during aging. Furthermore, we assessed the relationship between distinct longitudinal alterations in white matter integrity and cognition. White matter microstructural properties were estimated from diffusion magnetic resonance imaging, and cognitive status characterized from extensive neurocognitive testing. The same individuals were evaluated at two timepoints, with a mean interval time of 52.8 months (SD = 7.24) between first and last assessment. Our results show that age is associated with a decline in cognitive performance and a degradation in white matter integrity. Additionally, significant associations were found between diffusion measures and different cognitive dimensions (memory, executive function and general cognition). Overall, these results suggest that age-related cognitive decline is related to white matter alterations, and thus give support to the “disconnected hypothesis” of the aging brain.