The in-situ neutron diffraction technique, in combination with both the full-field crystal elasto-viscoplastic finite element model and microstructural characterization, was used to study the deformation-induced damage anisotropy in a commercial Al alloy, subjected to uniaxial tensile and cyclic loading. The simulations capture well the crystallographic-orientation-dependent lattice strain behavior. The hard grains, e.g. those orientated with the <111> and <422> orientations parallel with the uniaxial loading direction (LD), feature large Taylor factors and seem more prone to form damage-related band structures. Their effective elastic moduli decrease with the accumulation of damage, which are different from the soft grains orientated with the <200> orientation along the LD. Correlation between the distribution of voids and that of the residual lattice strain developed after failure may exist. The maximum tensile type residual lattice strain observed after failure may be resulted from the band structure formed in the hardest <111> grains. It was revealed that the band structure triggered by the hard particles could be one of sources of damage. In addition, while the specimen was obviously damaged, a fast stress relief was evidenced after unloading from the tension, especially at the beginning of unloading. Our present investigations provide a novel method for exploring the damage mechanisms of polycrystalline materials during plastic deformation.