Aims. According to previously published Doppler images of the magnetically active primary giant component of the RS CVn binary II Peg, the surface of the star was dominated by one single active longitude that was clearly drifting in the rotational frame of the binary system during 1994-2002; later imaging for 2004-2010, however, showed decreased and chaotic spot activity, with no signs of the drift pattern. Here we set out to investigate from a more extensive photometric dataset whether this drift is a persistent phenomenon, in which case it could be caused either by an azimuthal dynamo wave or be an indication that the binary system's orbital synchronization is still incomplete. On a differentially rotating stellar surface, spot structures preferentially on a certain latitude band could also cause such a drift, the disruption of which could arise from the change of the preferred spot latitude. Methods. We analyzed the datasets using the carrier fit (CF) method, which is especially suitable for analyzing time series in which a fast clocking frequency (such as the rotation of the star) is modulated with a slower process (such as the stellar activity cycle). Results. We combined all collected photometric data into one single data set and analyzed it with the CF method. We confirm the previously published results that the spot activity has been dominated by one primary spotted region almost through the entire data set and also confirm a persistent, nearly linear drift. Disruptions of the linear trend and complicated phase behavior are also seen, but the period analysis reveals a rather stable periodicity with Pspot = 6d.71054 ± 0 d.00005. After removing the linear trend from the data, we identified several abrupt phase jumps, three of which are analyzed in more detail with the CF method. These phase jumps closely resemble what is called a flip-flop event, but the new spot configurations do not persist for longer than a few months in most cases. Conclusions. There is some evidence that the regular drift without phase jumps is related to the high state, while the complex phase behavior and disrupted drift pattern are related to the low state of magnetic activity. The most natural explanation of the drift is weak anti-solar (pole rotating faster than the equator) differential rotation with a coefficient k ≈ 0.002 combined with the preferred latitude of the spot structure.