We develop a general microscopic theory describing the phonon decoherence of quantum dots and indistinguishability of the emitted photons in photonic structures. The coherence is found to depend fundamentally on the dimensionality of the structure resulting in vastly different performance for quantum dots embedded in a nanocavity (0D), waveguide (1D), slab (2D), or bulk medium (3D). In bulk, we find a striking temperature dependence of the dephasing rate scaling as T11 implying that phonons are effectively "frozen out" for T4 K. The phonon density of states is strongly modified in 1D and 2D structures leading to a linear temperature scaling for the dephasing strength. The resulting impact on the photon indistinguishability can be important even at sub-Kelvin temperatures. Our findings provide a comprehensive understanding of the fundamental limits to photon indistinguishability in photonic structures.