| As the WiMAX technology was approved as a 3G standard by International Telecommunications Union (ITU) in 2007 and IEEE 802.16m was acknowledged as the candidate technology of the 4G standardization in October 2009, WiMAX has become the most concerned wireless communications technology in recent years. To highlight the importance of security, IEEE 802.16 standard adds a security sublayer in the media access control layer. Since the Privacy Key Management (PKM) protocol is the core of the security sublayer, which is also directly related to the negotiation and distribution of the key between base station and subscriber station, and the authorization of service access etc., this work makes a comparative deep research on the PKM protocol in WiMAX network. The main results are as following:(1)The PKM Protocol in the newest IEEE 802.16j 2009 is deeply analyzed. To solve the security issues, backward and forward secrecy and key forgery, of Multicast and Broadcast Rekeying Algorithm (MBRA), we propose an ID based secure MBRA, named as SMBRA. Theoretical analysis and simulation results indicate that SMBRA is much more efficient and adequate than any other schemes, such as ELAPSE, especially in a large group.(2)We show that the newest fault tolerant group key agreement protocol, EGKA, proposed by Zhao et al. fails to achieve fault tolerant in several non trivial cases. To remedy these security flaws, we propose an improved EGKA protocol, named as EGKA+, which covers different key attack and the message tampering attack existing in EGKA protocol.(3)We point out several security issues, such as the key agreement blocking and Denial of Service (DoS) attacks, of the PKM protocol in IEEE 802.16m. To address the existing security issues, we propose an enhanced key management protocol, named as EKMP. EKMP is provably secure in the Canetti Krawczyk model and satisfies the correlative security attributes.(4)Inspired by the identity card, we propose a ticket based fast handover authentication mechanism. When the Mobile Station (MS) moves from the service Base station (BS) to a target BS, it can show its ticket to the target BS and this BS can authenticate the MS without communicating with any other third party (e.g., previous BS). Our analysis shows that the proposed scheme satisfies the security requirements in handover authentication semantics and provides robust efficiency in terms of communication overhead and computational cost. Thus it is well suited for efficient handover authentication with resource constrained MS.(5)We study the case of many correlated MSs moving together (For example, users taking the same vehicle such as a bus or a train) and design a Grouping Detection Algorithm (GDA) to group users based on the mobility correlation. Based on the GDA algorithm, we present a Group based Handover Authentication Protocol (GHAP) for correlated MSs using elliptic curve cryptography. The key idea of GHAP is that all the handover group members’security contexts are one time transmited to the target BS using the Security Context Transfer (SCT) method. Therefore, the handover performance is improved significantly. |