Receiving a tender caress from a caregiver or spouse reduces stress and promotes emotional well-being, but receiving the same caress from a stranger makes us feel uncomfortable. According to recent neurophysiological findings, we not only react differently to the invited versus uninvited touch but also perceive the touch differently depending on context. A virtual reality experiment was conducted to investigate whether individual differences regarding behavioral inhibition system (BIS) and gender contribute to this affective touch perception. Touch perception was measured directly using self-reports and indirectly using the touch-related orienting response. The results showed that touch perception depended on the emotional expression of the virtual agents. High-arousal approach-related (happiness, anger) and avoidance-related (fear) expressions increased self-reported touch intensity, while happiness reduced the orienting response to touch. Moreover, interpersonal differences in behavioral inhibition and gender played distinct roles: BIS sensitivity in males was associated with stronger affective touch perception, particularly with high-arousal emotions whereas in females BIS sensitivity did not affect touch perception. The results suggest that individual differences that are related to preferences regarding tactile communication also determine how touch is perceived.