In recent times, additive manufacturing (AM) has proven to be an indispensable technique for processing complex 3D parts because of the versatility and ease of fabrication it offers. However, the generated microstructures show a high degree of complexity due to the complex solidification process of the melt pool. In this study, micromechanical modeling is applied to gain deeper insight into the influence of defects on plasticity and damage of 316L stainless steel specimens produced by a laser powder bed fusion (L-PBF) process. With the statistical data obtained from microstructure characterization, the complex AM microstructures are modeled by a synthetic microstructure generation tool. A damage model in combination with an element deletion technique is implemented into a nonlocal crystal plasticity model to describe anisotropic mechanical behavior, including damage evolution. The element deletion technique is applied to effectively model the growth and coalescence of microstructural pores as described by a damage parameter. Numerical simulations show that the shape of the pores not only affects the yielding and hardening behavior but also influences the porosity evolution itself.