The recent discovery that impurity atoms in crystals can be manipulated with focused electron irradiation has opened novel perspectives for top-down atomic engineering. These achievements have been enabled by advances not only in electron optics and microscope stability but also in the preparation of suitable materials with impurity elements incorporated via ion and electron-beam irradiation or chemical means. Here it is shown that silicon heteroatoms introduced via plasma irradiation into the lattice of single-walled carbon nanotubes (SWCNTs) can be manipulated using a focused 55–60 keV electron probe aimed at neighboring carbon sites. Moving the silicon atom mainly along the longitudinal axis of large 2.7 nm diameter tubes, more than 90 controlled lattice jumps are recorded and the relevant displacement cross sections are estimated. Molecular dynamics simulations show that even in 2 nm diameter SWCNTs, the threshold energies for out-of-plane dynamics are different than in graphene, and depend on the orientation of the silicon-carbon bond with respect to the electron beam as well as the local bonding of the displaced carbon atom and its neighbors. Atomic-level engineering of SWCNTs where the electron wave functions are more strictly confined than in 2D materials may enable the fabrication of tunable electronic resonators and other devices.