Changes in extracellular matrix stiffness impact a variety of biological processes including cancer progression. However, cells also actively remodel the matrices they interact with, dynamically altering the matrix mechanics they respond to. Further, cells not only react to matrix stiffness, but also have a distinct reaction to matrix viscoelasticity. The impact of cell-driven matrix remodeling on matrix stiffness and viscoelasticity at the microscale remains unclear, as existing methods to measure mechanics are largely at the bulk scale or probe only the surface of matrices, and focus on stiffness. Yet, establishing the impact of the matrix remodeling at the microscale is crucial to obtaining an understanding of mechanotransduction in biological matrices, and biological matrices are not just elastic, but are viscoelastic. Here, we advanced magnetic probe-based microrheology to overcome its previous limitations in measuring viscoelasticity at the cell-size-scale spatial resolution within 3D cell cultures that have tissue-relevant stiffness levels up to a Young’s modulus of 0.5 kPa. Our magnetic microrheometers exert controlled magnetic forces on magnetic microprobes within reconstituted extracellular matrices and detect microprobe displacement responses to measure matrix viscoelasticity and determine the frequency-dependent shear modulus (stiffness), the loss tangent, and spatial heterogeneity. We applied these tools to investigate how microscale viscoelasticity of collagen matrices is altered by fibroblast cells as they contract collagen gels, a process studied extensively at the macroscale. Interestingly, we found that fibroblasts first soften the matrix locally over the first 32 hours of culture, and then progressively stiffen the matrix thereafter. Fibroblast activity also progressively increased the matrix loss tangent. We confirmed that the softening is caused by matrix-metalloproteinase-mediated collagen degradation, whereas stiffening is associated with local alignment and densification of collagen fibers around the fibroblasts. This work paves the way for the use of measurement systems that quantify microscale viscoelasticity within 3D cell cultures for studies of cell–matrix interactions in cancer progression and other areas.