Rotary vane steering gear has the advantages of compact structure, high mechanical efficiency, and easy installation, and is widely used in ships. The existing rotary vane steering gear is a single-layer hydraulic swing cylinder, and the rotation range of the rudder blade is restricted by the structure of the swing cylinder and has saturated nonlinearity. In addition, the phenomena of rudder impingement, lag and running caused by hydrodynamic interference seriously affect the ship’s course control and rudder anti-roll effect. Aiming at the above problems, a new steering principle of compound structure swing cylinder is proposed based on the mathematical model of direct drive electro-hydraulic servo rotary vane actuator and the analysis of the interference of hydrodynamic force on rudder angle. The double-layer structure is adopted for the compound swing cylinder, the inner layer is rudder driving cylinder, the outer layer is torque decoupling cylinder. The inner and outer rotors rotate in the same direction, which can increase the working range of the rudder blade. Meanwhile, the torque decoupling cylinder rotor outputs boost torque acting on the rudder drive cylinder rotor, which can offset the load torque generated by the hydrodynamic force on the drive cylinder rotor. Improve the rotation accuracy of the rudder blade and solve the problem of force-position coupling in the movement of the rudder. The reverse rotation of the inner and outer rotors can make the steering gear brake and change direction in time, and improve the steering performance. The compound swing cylinder is used for the vane steering gear. The steering gear system has a large stability margin, with amplitude margin of 45.3 dB and phase margin of 99.2°, which meets the design index of servo system. Simulation analysis shows that, compared with the single-layer swing cylinder plus control strategy, the compound swing cylinder has faster response speed and no overshoot, the speed of reaching steady state increased by about 36%, the steady-state error is maintained within ±0.05° under external load interference. When tracking a slope signal with a slope of 0.01°/s, the steering accuracy can be maintained within the ±0.03° position error band, which has high position control accuracy.