Replacing conventional buses with electric buses is in line with the concept of sustainable development. However, electric buses have the disadvantages of short driving range and high purchase price. Many cities must implement a semi-electrification strategy for bus routes. In this paper, a bi-level, multi-objective programming model is established for the collaborative scheduling problem of vehicles and drivers on a bus route served by the mixed bus fleet. The upper-layer model minimizes the operation cost and economic cost of carbon emission to optimize the vehicle and charging scheme; while the lower-layer model tries to optimize the crew-scheduling scheme with the objective of minimizing driver wages and maximizing the degree of bus-driver specificity, considering the impact of drivers’ labor restrictions. Then, the improved multi-objective particle swarm algorithm based on an ε-constraint processing mechanism is used to solve the problem. Finally, an actual bus route is taken as an example to verify the effectiveness of the model. The results show that the established model can reduce the impact of unbalanced vehicle scheduling in mixed fleets on crew scheduling, ensure the degree of driver–bus specificity to standardize operation, and save the operation cost and driver wage.