In the present combined DTI/fMRI study we investigated adaptive plasticity of neural networks involved in controlling spatial and nonspatial auditory working memory in the early blind (EB). In both EB and sighted controls (SC), fractional anisotropy (FA) within the right inferior longitudinal fasciculus correlated positively with accuracy in a one-back sound localization but not sound identification task. The neural tracts passing through the cluster of significant correlation connected auditory and “visual” areas in the right hemisphere. Activity in these areas during both sound localization and identification correlated with FA within the anterior corpus callosum, anterior thalamic radiation, and inferior fronto-occipital fasciculus. In EB, FA in these structures correlated positively with activity in both auditory and “visual” areas, whereas FA in SC correlated positively with activity in auditory and negatively with activity in visual areas. The results indicate that frontal white matter conveys cross-modal suppression of occipital areas in SC, while it mediates coactivation of auditory and reorganized “visual” cortex in EB.