The lattice thermal conductivity of silicon clathrate framework Si23 and two Zintl clathrates, [Si19P4]Cl4 and Na4[Al4Si19], is investigated by using an iterative solution of the linearized Boltzmann transport equation in conjunction with ab initio lattice dynamical techniques. At 300 K, the lattice thermal conductivities for Si23, [Si19P4]Cl4, and Na4[Al4Si19] were found to be 43 W/(m K), 25 W/(m K), and 2 W/(m K), respectively. In the case of Na4[Al4Si19], the order-of-magnitude reduction in the lattice thermal conductivity was found to be mostly due to relaxation times and group velocities differing from Si23 and [Si19P4]Cl4. The difference in the relaxation times and group velocities arises primarily due to the phonon spectrum at low frequencies, resulting eventually from the differences in the second-order interatomic force constants (IFCs). The obtained third-order IFCs were rather similar for all materials considered here. The present findings are similar to those obtained earlier for some skutterudites. The predicted lattice thermal conductivity of Na4[Al4Si19] is in line with the experimentally measured thermal conductivity of recently synthesized type-I Zintl clathrate Na8[Al8Si38] (polycrystalline samples).