During joint actions, people typically adjust their own actions according to the ongoing actions of the partner, which implies that the interaction modulates the behavior of both participants. However, the neural substrates of such mutual adaptation are still poorly understood. Here, we set out to identify the kinematics-related brain activity of leaders and followers performing hand actions. Sixteen participants as 8 pairs performed continuous, repetitive right-hand opening and closing actions with ~3-s cycles in a leader-follower task. Subjects played each role for 5 min. Magnetoencephalographic (MEG) brain signals were recorded simultaneously from both partners with a dual-MEG setup, and hand kinematics was monitored with accelerometers. Modulation index, a cross-frequency coupling measure, was computed between the hand acceleration and the MEG signals in the alpha (7-13 Hz) and beta (13-25 Hz) bands. Regardless of the participants' role, the strongest alpha and beta modulations occurred bilaterally in the sensorimotor cortices. In the occipital region, beta modulation was stronger in followers than leaders; these oscillations originated, according to beamformer source reconstructions, in early visual cortices. Despite differences in the modulation indices, alpha and beta power did not differ between the conditions. Our results indicate that the beta modulation in the early visual cortices depends on the subject's role as a follower or leader in a joint hand-action task. This finding could reflect the different strategies employed by leaders and followers in integrating kinematics-related visual information to control their own actions.