Zirconium disulfide (ZrS2) and zirconium diselenide (ZrSe2) are promising materials for future optoelectronics due to indirect band gaps in the visible and near-infrared (NIR) spectral regions. Alloying these materials to produce ZrSxSe2-x (x = 0 horizontal ellipsis 2) would provide continuous control over key optical and electronic parameters required for device engineering. Here, we present a comprehensive analysis of the phonons and excitons in ZrSxSe2-x using low-temperature Raman spectroscopy and room-temperature spectroscopic ellipsometry (SE) measurements. We extract the Raman-active vibrational mode frequencies and find that they compare favorably with density functional theory (DFT) calculations. Our simulations and polarization-resolved measurements demonstrate that substitutional doping renders infrared (IR) modes to be Raman-active. This leads to a Raman spectrum dominated by nominally IR phonons, a phenomenon that originates from the large ionicity of the ZrSxSe2-x bonds. SE measurements of the complex refractive index quantify the blue-shift of direct, allowed exciton transitions with increasing S content, and we find strong light-matter interactions with low optical loss in the NIR. Correlating these data with DFT allows for an estimation of the Gamma-point exciton binding energy at room temperature. This study illustrates the large effects of alloying on ZrSxSe2-x and lays the foundation for future applications of this material.