Abstract:We propose a density-matrix-formalism based scheme to study polarization mode dispersion (PMD) monitoring and compensation in optical fiber communication systems. Compared to traditional monitoring and compensation schemes based on the PMD vector in the Stokes space, the scheme we proposed requires no auxiliary matrices and can be handily extended to any higher-dimensional modal spaces, which is advantageous in mode-division multi-plexing (MDM) systems. A 28GBaud polarization division multiplexing quadrature phase-shift keying (PDM-QPSK) coherent simulation system is built to demonstrate that our scheme can implement the monitoring and compen-sation of 170 ps large differential-group-delay (DGD) that far exceeds the typical DGDs in practical optical com-munication systems. The results verify the effectiveness of the density-matrix-formalism based scheme in PMD monitoring and compensation, thus pave the way for further applications of the scheme in more general MDM optical communication systems.