Machine Learning Enabled Nonlinear Equalization Technique For Fiber Optical Communication System

In view of the rapid development of fiber optical communication systems with ultra-large capacity,ultra-high speed bit-rate,and ultra-long distance,both short-and medium-reach intensity modulation and direct detection(IM-DD)transmission systems and long-reach digital coherent transmission systems will suffer from more complex transmission impairments.Exploring the machine learning(ML)techniques to achieve more efficient nonlinear equalization(NLE)for fiber optical communication systems is vital to compensate various transmission impairments,enhance the transmission capacity,and provide a sustainable solution for fiber optical communication systems.To realize a reliable equalization performance evaluation,the overfitting effect is comprehensively investigated,when the pseudo-random binary bit sequences(PRBS)and the artificial neural network(ANN)are used for the signal generation and the NLE,respectively,leading to a bit-error ratio(BER)performance overestimation.Both the mathematical origin and the transmission evolution of the overfitting effect are demonstrated.To mitigate the overfitting effect,both the Mersenne twister pseudo-random sequence(MTRS)and Pruned PRBS(P-PRBS)are proposed to provide a reliable benchmark of BER evaluation.In order to achieve the NLE with high performance and low complexity,two research methods are further considered,including ML-enabled direct NLEs and ML-assisted optimization of traditional NLEs.For both IM-DD systems and digital coherent transmission systems,the efficiency of ANN-NLE is investigated.Then the Volterra filter equalizer(VFE)of IM-DD transmission and the perturbation nonlinear compensation(PNC)of digital coherent transmission are optimized by ML schemes.The main innovative research outcomes are summarized as follows.(1)For the overfitting effect of ANN-NLE with PRBS symbols,it is verified that ANN can learn the PRBS generation and bit-to-symbol mapping rules by increasing the weights of input neurons at specific locations.Moreover,it is experimentally discovered that the pulse spreading induced by both dispersion and fiber nonlinearity can further worsen the overfitting effect through the fiber optical transmission.P-PRBS dataset can alleviate the overfitting effect,while MTRS signals can completely avoid the overfitting effect.Therefore,MTRS can be used as a reliable source for the subsequent research.(2)To address the performance and complexity challenges of NLE in the IM-DD transmission systems,it is identified that there occurs an approximately linear relationship between the performance of the ANN-NLE and the calculation complexity represented by the number of multipliers.Under the same BER threshold,the complexity of ANN-NLE is about three times higher than that of VFE.To further improve the equalization efficiency of VFE,a weighted principal component analysis optimized VFE scheme is proposed.Almost40% and 60% of the number of VFE taps are experimentally saved for the dispersion unmanaged link and the dispersion managed link,respectively.(3)For digital coherent transmission systems,it is found that the ANN-NLE will result in a square-shaped constellation diagram,leading to a performance overestimation of signal-to-noise ratio.Alternatively,such overestimation can be mitigated by modifying the ANN output as the nonlinear residual.To optimize the efficiency of PNC,the hard-decision operator generation scheme is proposed to relieve the required bits of multipliers.Meanwhile,both the hyperbolic scheme for screening operators and the recursive least squares algorithm for training the weights are proposed to avoid the sensitivity of the fiber channel parameters.Further assisted by the ML techniques,the principal component analysis scheme can save about 40% of the number of PNC operators in the single-wave transmission channel,and the transfer learning scheme can improve the PNC performance of wavelength division multiplexing transmission by 0.2 d B.