As recent experiments demonstrate, the inner shells of multiwalled carbon nanotubes are more sensitive to electron irradiation than the outer shells. To understand the origin of such counterintuitive behavior, we employ a density-functional-theory based tight-binding method and calculate the displacement threshold energies for carbon atoms in single-walled nanotubes with different diameters and chiralities. We show that the displacement energy and the defect production rate strongly depend on the diameter of the nanotube and its chirality, with the displacement energy being lower, but saturating towards the value for graphite when the tube diameter increases. This implies that the threshold electron energies to produce damage in nanotubes with diameters smaller than 1nm are less than the commonly accepted value for graphitic nanoparticles. We also calculate the displacement energies for carbon atoms near defects and show that if a single vacancy is formed, it will likely be transformed to a double vacancy, as the nanotube atomic network with double vacancies has no energetically unfavorable undercoordinated atoms.