The concept of surface free energy was utilized to assess the effect of binder aging, mineral fillers, and aggregate type on the moisture damage resistance of asphalt concrete. A known methodology was used to quantify the dry and wet adhesive bond energies between four types of asphalt mastics and two types of aggregates. Furthermore, the effect of binder aging was evaluated by testing asphalt mastics with different aging conditions: non-aged, RTFO-aged (short-term aging), and PAV-aged (long-term aging). The compatibility ratio (CR) was then quantified for each mastic-aggregate combination and used as an indicator of moisture resistance of asphalt mixtures. Results confirmed that granite aggregates are more prone to moisture damage than limestone aggregates. Furthermore, cement and lime mastics enhanced the resistance of the system to moisture damage compared to limestone filler, which resulted in higher moisture susceptibility. Finally, results also revealed that aged mastics lead to lower CR, which suggests that moisture susceptibility of asphalt mixtures increases as the pavement ages.