Multi-Agent Deep Reinforcement Learning for Handover Management in Massive Industrial Internet of Things Networks
Main Article Content
Abstract
The industrial internet of things (IioT) is considered one of theIapplications in the fifth generation (5G) networks. In this application, users’ high mobility in a typical industrial scenario needs high reliability. The high mobility creates frequent handover, creating extra control signalling to a new base station (BS). The users’ congestion to the new BSs can lead to an outage. In this paper, we investigate how to manage the handover of users to improve reliability in a high-mobility scenario using deep learning. We first use an offline centralized algorithm to create labels for user association to a BS that is done without adding handover coefficient. Then, we train the neural network and use the trained parameter to make the multi-agent deep reinforcement learning (RL) learns better. This is done to avoid long iterative methods in reinforcement learning. The results show that our method can outperform the offline centralized algorithm by 40% when the handover coefficient increases.
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Copyright on any article is retained by the author(s).
- Author grant the journal, right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work’s authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal’s published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work.
- The article and any associated published material is distributed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License
References
A. Gupta and R. K. Jha. 2015. “A Survey of 5G Network: Architecture and Emerging Technologies.” IEEE Access 3:1206–32. doi: 10.1109/ACCESS.2015.2461602.
B. Singh, O. Tirkkonen, Z. Li, and M. A. Uusitalo. 2018. “Contention-Based Access for Ultra-Reliable Low Latency Uplink Transmissions.” IEEE Wireless Communications Letters 7(2):182–85. doi: 10.1109/LWC.2017.2763594.
C. She, C. Sun, Z. Gu, Y. Li, C. Yang, H. V. Poor, and B. Vucetic. 2021. “A Tutorial on Ultrareliable and Low-Latency Communications in 6G: Integrating Domain Knowledge Into Deep Learning.” Proceedings of the IEEE 109(3):204–46. doi: 10.1109/JPROC.2021.3053601.
Chen, Zhaohui, Zhaoyang Luo, Xiaohui Duan, and Lianming Zhang. 2020. “Terminal Handover in Software-Defined WLANs.” EURASIP Journal on Wireless Communications and Networking 2020(1):68. doi: 10.1186/s13638-020-01681-w.
D. Guo, L. Tang, X. Zhang, and Y. -C. Liang. 2020. “Joint Optimization of Handover Control and Power Allocation Based on Multi-Agent Deep Reinforcement Learning.” IEEE Transactions on Vehicular Technology 69(11):13124–38. doi: 10.1109/TVT.2020.3020400.
D. Liu, L. Wang, Y. Chen, M. Elkashlan, K. -K. Wong, R. Schober, and L. Hanzo. 2016. “User Association in 5G Networks: A Survey and an Outlook.” IEEE Communications Surveys & Tutorials 18(2):1018–44. doi: 10.1109/COMST.2016.2516538.
H. Xu, D. Li, M. Liu, G. Han, W. Huang, and C. Xu. 2020. “QoE-Driven Intelligent Handover for User-Centric Mobile Satellite Networks.” IEEE Transactions on Vehicular Technology 69(9):10127–39. doi: 10.1109/TVT.2020.3000908.
Kingma, Diederik P., and Jimmy Ba. 2015. “Adam: A Method for Stochastic Optimization.” in 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Mnih, Volodymyr, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. 2015. “Human-Level Control through Deep Reinforcement Learning.” Nature 518(7540):529–33. doi: 10.1038/nature14236.
N. Raharya, C. She, W. Hardjawana, and B. Vucetic. 2021. “Deep Learning for Distributed User Association in Massive Industrial IoT Networks.” Pp. 1–6 in 2021 IEEE Wireless Communications and Networking Conference (WCNC).
N. Zhao, Y. -C. Liang, D. Niyato, Y. Pei, M. Wu, and Y. Jiang. 2019. “Deep Reinforcement Learning for User Association and Resource Allocation in Heterogeneous Cellular Networks.” IEEE Transactions on Wireless Communications 18(11):5141–52. doi: 10.1109/TWC.2019.2933417.
Q. Ye, B. Rong, Y. Chen, M. Al-Shalash, C. Caramanis, and J. G. Andrews. 2013. “User Association for Load Balancing in Heterogeneous Cellular Networks.” IEEE Transactions on Wireless Communications 12(6):2706–16. doi: 10.1109/TWC.2013.040413.120676.
Raharya, Naufan. 2021. “Machine Learning for Massive Connections in Wireless Networks.”
S. Tekinay and B. Jabbari. 1991. “Handover and Channel Assignment in Mobile Cellular Networks.” IEEE Communications Magazine 29(11):42–46. doi: 10.1109/35.109664.
Third Generation Partnership Project (3GPP). 2014. Spatial Channel Model for Multiple Input Multiple Output (MIMO) Simulations TR 25.996 V.12.0.0. Standard.
Third Generation Partnership Project (3GPP). n.d. Study on New Radio (NR) Access Technology; Physical Layer Aspects (Release 14) TR 38.802 V.2.0.0. Standard. TR 38.802 V.2.0.0.
V. Yajnanarayana, H. Rydén, and L. Hévizi. 2020. “5G Handover Using Reinforcement Learning.” Pp. 349–54 in 2020 IEEE 3rd 5G World Forum (5GWF).
W. Sun, L. Wang, J. Liu, N. Kato, and Y. Zhang. 2021. “Movement Aware CoMP Handover in Heterogeneous Ultra-Dense Networks.” IEEE Transactions on Communications 69(1):340–52. doi: 10.1109/TCOMM.2020.3019388.