We propose the use of adaptive optimized irregular constellation to enhance the performance of full duplex (FD) wireless radio systems which suffer from residual self-interference (RSI). Specifically, an error performance expression which takes into account the RSI level and signal-to-noise ratio is first derived for a single FD relay link. During the search for optimal constellation for a given parameter set, there are no predefined assumptions on symbol point locations. The optimization framework yields adaptive irregular constellations depending on RSI levels and power budget values. It is shown that the proposed adaptive transmission along with optimized irregular constellations achieves better error performance than the conventional M-QAM constellations and other M-ary improper Gaussian signaling (IGS) constellations.