In order to continue to ensure the sustainability of mobile communication services over the next decade and to meet the business and consumer demands, fifth genera-tion (5G) mobile communication services is expected to be rolled out by 2020. One of the major requirements for 5G networks is the significant spectral efficiency (SE) enhancement compared to fourth generation (4G) as the anticipated exponential increase in the volume of mobile data traffic is huge, for example, at least 1,000-fold in the 2020s compared to 2010. In particular, the peak data rate in 5G should be 10–20 Gbps that is 10–20 times the peak data rate in 4G, and the user experienced data rate should be 1 Gbps (100 times the user experienced data rate in 4G). In addition, the rapid development of Mobile Internet and the Internet of Things (IoT) exponentially accelerates the demands for high data rate applications, including high-quality video streaming, social networking, and machine-to-machine communications.
In cellular network, the design of radio access technology, in general, and multiple access technique, in particular, are one of the most important aspects in improving the system capacity. Multiple access techniques are usually categorized into two orthogonal and nonorthogonal approaches [1]. In orthogonal approaches