Generally, the noise transmitted in the optical path mainly includes the influence of optical power, external vibration interference, the intrinsic noise of the optical path, and the influence of thermal motion caused by environmental temperature changes. The optical fiber is extremely sensitive to environmental interference, and any environmental disturbance will bring corresponding phase errors. At the same time, if the signal is preprocessed and filtered, it is easy to cause some small information to be lost and increase reconstruction errors. Aiming at the problem of internal common mode noise affecting measurement accuracy and increasing reconstruction error in micron-level vibration measurement, a differential measurement system based on all-fiber Fabry–Perot (F–P) interference is established. Firstly, according to the phase equation of the F–P interference principle and the 2×2 single-mode fiber coupler during the transmission process, the through arm and the coupling arm have a phase difference of 90°. It is deduced that the signals received by the two output detectors are equal in amplitude and opposite in direct. Then the difference between the two signals can increase the overall signal amplitude, weaken the influence of common mode noise on the signal, and enhance the signal quality. Through the reconstruction displacement analysis of the interference signal, the experimental results show that the reconstruction error of the synthesized signal is reduced by 0.4% in the 2 μm sinusoidal vibration of the mirror. Under 1.4 μm vibration measurement on rough surface, the reconstruction error is reduced by 2.8%. The experimental results also show that the differential structure can reduce the vibration measurement reconstruction error, and the reconstructed waveform is smoother.