Normal aging is characterized by structural and functional changes in the brain contributing to cognitive decline. Structural connectivity (SC) describes the anatomical backbone linking distinct functional subunits of the brain and disruption of this communication is thought to be one of the potential contributors for the age-related deterioration observed in cognition. Several studies already explored brain network's reorganization during aging, but most focused on average connectivity of the whole-brain or in specific networks, such as the resting-state networks. Here, we aimed to characterize longitudinal changes of white matter (WM) structural brain networks, through the identification of sub-networks with significantly altered connectivity along time. Then, we tested associations between longitudinal changes in network connectivity and cognition. We also assessed longitudinal changes in topological properties of the networks. For this, older adults were evaluated at two timepoints, with a mean interval time of 52.8 months (SD = 7.24). WM structural networks were derived from diffusion magnetic resonance imaging, and cognitive status from neurocognitive testing. Our results show age-related changes in brain SC, characterized by both decreases and increases in connectivity weight. Interestingly, decreases occur in intra-hemispheric connections formed mainly by association fibers, while increases occur mostly in inter-hemispheric connections and involve association, commissural, and projection fibers, supporting the last-in-first-out hypothesis. Regarding topology, two hubs were lost, alongside with a decrease in connector-hub inter-modular connectivity, reflecting reduced integration. Simultaneously, there was an increase in the number of provincial hubs, suggesting increased segregation. Overall, these results confirm that aging triggers a reorganization of the brain structural network.