We mainly investigate a coupled system of the generalized nonlinear Schrodinger equation and the Maxwell-Bloch equations which describes the wave propagation in an erbium-doped nonlinear fiber with higher-order nonlinear effects including the forth-order dispersion and quin-tic non-Kerr nonlinearity. We provide the one-fold Darbox transformation of this system and construct the determinant representation Tn of the n-fold Darboux transformation. By using the Tn, the (E[n, p[n],η[n]) of the GNLS-MB system are generated from the seed solution (E, p, η). We construct the soliton solutions of the GNLS-MB system by using zero seed solutions. The first-order Positon solution of the GNLS-MB system is given by degeneration of eigenvalues in double-soliton. Then we get the breather solutions of this system from periodic seed solutions. Furthermore, we construct the first-order rogue wave by a limit of the first-order breather, and construct the determinant representation of the nth-order rogue waves by Taylor expansion-s. The hybrid solution of breathers and rogue waves are given by the nonlinear superposition of them. According to the explicit form of the first-order rogue wave, we show a remarkable rotation and compression effects on rogue wave due to higher-order nonlinear terms. |