| Multiple-input Multiple-output (MIMO) technology can achieve both high spectral efficiency and transmission reliability by transmitting and receiving multiple data streams concurrently. Therefore, MIMO technology has attracted a lot of attention by researchers over years. The existing eight antenna ports in LTE-A system (Full-dimension MIMO system with 12/16 antenna ports is proposed in 3GPP Release 13) cannot satisfy the constantly increasing demands for higher data rates. As the one of the key technologies for the next generation wireless communication system, Massive MIMO (also known as large-scale MIMO) technology applying tens to hundreds of antennas can achieve huge improvements in the system capacity and energy efficiency. However, with the increased number of antennas,the spatial channel characteristics will be more complicated, and it will pose a huge challenge for the receiver design of signals.Firstly, we discuss three kinds of the received signal processing structure, i.e., independent signal detector and channel decoder, Turbo structure based signal detector and channel decoder, and associated signal detector and channel decoder. Soft-input soft-output (SISO) sphere decoding is applied to joint iterative detection and decoding (Turbo receiver) to offer good bit error rate (BER) performance but at high complexity. To reduce this computational complexity, we propose a novel hybrid enumeration strategy that dynamically determines the candidate list.Moreover, this strategy uses a new node enumeration based on concentric circles. Specifically, points in an MQAM constellation diagram are divided into several subsets, and the visiting order among subsets is predefined by the concentric circle locations. Simulation results demonstrate that the proposed enumeration strategy outperform previous approach, while keeping a low complexity.Secondly, we discuss linear detection algorithms consisting of the truncated Neumann series (NS) expansion and iteration based linear detector for massive MIMO system, such as Jacobi, Richardson, Gauss-Seidel (GS), successive over relaxation (SOR) and symmetric successive over relaxation (SSOR). These iterative detection algorithms based on minimum mean square error (MMSE) criterion are applied to massive MIMO system owing to channel hardening. However, one of huge challenges for linear detection algorithms in massive MIMO system is the full matrix inversion operation, which brings extremely high computational complexity. In this thesis, we proposed a modified Jacobi iteration based MMSE detection algorithm avoiding matrix inversion operation. The initial solution vector in the iterative algorithm can be calculated by the truncated Neumann series (NS) expansion, which can obtain the optimal solution vector in a faster convergence rate.Finally, a low-complexity near-ML signal detection algorithm,termed hybrid constellation diagram based sphere decoding (HCD-SD) is proposed for generalized spatial modulation. Specifically, the detection is performed in two steps: the first step is to estimate the initial transmit antenna combination (TAC) by MMSE equalization based on hybrid constellation diagram; the second step is to calibrate the TAC by sorting the TAC candidates based on Euclidean distance and detect the symbol by sphere decoding. |