In project portfolio selection, the aim is to choose projects which are expected to offer most value and satisfy relevant risk and other constraints. In this study, we show that uncertainties about how much value the projects will offer, combined with the fact that only a subset of the proposed projects will be selected, lead to inaccurate risk estimates about the aggregate value provided by the selected project portfolio. In particular, when downside risks are measured in terms of lower percentiles of the distribution of portfolio value, these risk estimates will exhibit a systematic bias. For deriving unbiased risk estimates, we present a calibration framework in which the required calibration can be presented in closed-form in some cases or, more generally, derived by using Monte Carlo simulation to study a large number of project selection decisions. We also show that when the decision must comply with risk constraints, the introduction of tighter (more demanding) risk constraints can counterintuitively aggravate the underestimation of risks. Finally, we present how the calibrated risk estimates can be employed to align the portfolio with the decision maker's risk preferences while eliminating systematic biases in risk estimates.