We identify the deviation from the straight-line error (DSLE) - i.e., the spurious nonlinearity of the total energy as a function of fractional particle number - as the main source for the discrepancy between experimental vertical ionization energies and theoretical quasiparticle energies, as obtained from the GW and GW+SOSEX approximations to many-body perturbation theory (MBPT). To check whether a DSLE is present in GW, we propose an indicator that only invokes observables at integer particle numbers. For self-consistent calculations, we show that GW suffers from a small DSLE. Conversely, for perturbative G0W0 and G0W0+SOSEX calculations the DSLE depends on the starting point. We exploit this starting point dependence to reduce (or completely eliminate) the DSLE. We find that the agreement with experiment increases as the DSLE reduces. DSLE-minimized schemes thus emerge as promising avenues for future developments in MBPT.