Deep Q-Learning for Two-Hop Communications of Drone Base Stations

Azade Fotouhi; Ming Ding; Mahbub Hassan

doi:10.3390/s21061960

Deep Q-Learning for Two-Hop Communications of Drone Base Stations

Sensors ◽

10.3390/s21061960 ◽

2021 ◽

Vol 21 (6) ◽

pp. 1960

Author(s):

Azade Fotouhi ◽

Ming Ding ◽

Mahbub Hassan

Keyword(s):

Degrees Of Freedom ◽

Network Performance ◽

Learning Model ◽

Base Stations ◽

Communication Model ◽

Complex Environments ◽

End User ◽

Q Learning ◽

Trajectory Simulation ◽

Target Environment

In this paper, we address the application of the flying Drone Base Stations (DBS) in order to improve the network performance. Given the high degrees of freedom of a DBS, it can change its position and adapt its trajectory according to the users movements and the target environment. A two-hop communication model, between an end-user and a macrocell through a DBS, is studied in this work. We propose Q-learning and Deep Q-learning based solutions to optimize the drone’s trajectory. Simulation results show that, by employing our proposed models, the drone can autonomously fly and adapts its mobility according to the users’ movements. Additionally, the Deep Q-learning model outperforms the Q-learning model and can be applied in more complex environments.

Download Full-text

Intelligent Interference Management in UAV-Based HetNets

Telecom ◽

10.3390/telecom2040027 ◽

2021 ◽

Vol 2 (4) ◽

pp. 472-488

Author(s):

Simran Singh ◽

Abhaykumar Kumbhar ◽

İsmail Güvenç ◽

Mihail L. Sichitiu

Keyword(s):

Cellular Networks ◽

Hot Spots ◽

Network Performance ◽

Interference Management ◽

Base Stations ◽

Computationally Efficient ◽

Q Learning ◽

Conventional Optimization ◽

Coverage Holes ◽

Mobile Base

Unmanned aerial vehicles (UAVs) can play a key role in meeting certain demands of cellular networks. UAVs can be used not only as user equipment (UE) in cellular networks but also as mobile base stations (BSs) wherein they can either augment conventional BSs by adapting their position to serve the changing traffic and connectivity demands or temporarily replace BSs that are damaged due to natural disasters. The flexibility of UAVs allows them to provide coverage to UEs in hot-spots, at cell-edges, in coverage holes, or regions with scarce cellular infrastructure. In this work, we study how UAV locations and other cellular parameters may be optimized in such scenarios to maximize the spectral efficiency (SE) of the network. We compare the performance of machine learning (ML) techniques with conventional optimization approaches. We found that, on an average, a double deep Q learning approach can achieve 93.46% of the optimal median SE and 95.83% of the optimal mean SE. A simple greedy approach, which tunes the parameters of each BS and UAV independently, performed very well in all the cases that we tested. These computationally efficient approaches can be utilized to enhance the network performance in existing cellular networks.

Download Full-text

Trajectory Design for the Aerial Base Stations to Improve Cellular Network Performance

IEEE Transactions on Vehicular Technology ◽

10.1109/tvt.2021.3049367 ◽

2021 ◽

pp. 1-1

Author(s):

Behzad Khamidehi ◽

Elvino Silveira Sousa

Keyword(s):

Cellular Network ◽

Network Performance ◽

Base Stations ◽

Trajectory Design

Download Full-text

A strategy learning model for autonomous agents based on classification

International Journal of Applied Mathematics and Computer Science ◽

10.1515/amcs-2015-0035 ◽

2015 ◽

Vol 25 (3) ◽

pp. 471-482 ◽

Cited By ~ 7

Author(s):

Bartłomiej Śnieżyński

Keyword(s):

Reinforcement Learning ◽

Supervised Learning ◽

Learning Process ◽

Autonomous Agents ◽

Good Alternative ◽

Learning Model ◽

Learning Method ◽

Complex Environments ◽

Agent Based ◽

Proposed Model

AbstractIn this paper we propose a strategy learning model for autonomous agents based on classification. In the literature, the most commonly used learning method in agent-based systems is reinforcement learning. In our opinion, classification can be considered a good alternative. This type of supervised learning can be used to generate a classifier that allows the agent to choose an appropriate action for execution. Experimental results show that this model can be successfully applied for strategy generation even if rewards are delayed. We compare the efficiency of the proposed model and reinforcement learning using the farmer-pest domain and configurations of various complexity. In complex environments, supervised learning can improve the performance of agents much faster that reinforcement learning. If an appropriate knowledge representation is used, the learned knowledge may be analyzed by humans, which allows tracking the learning process

Download Full-text

Research on Network Communication Model of Intelligent Ship Handling Simulator

Applied Mechanics and Materials ◽

10.4028/www.scientific.net/amm.97-98.787 ◽

2011 ◽

Vol 97-98 ◽

pp. 787-793 ◽

Cited By ~ 1

Author(s):

Shen Hua Yang ◽

Guo Quan Chen ◽

Xing Hua Wang ◽

Yue Bin Yang

Keyword(s):

Mathematical Model ◽

Degrees Of Freedom ◽

Communication Process ◽

Network Communication ◽

Communication Model ◽

Test Results ◽

Six Degrees Of Freedom ◽

Model Based ◽

Hydraulic Servo ◽

Ship Handling

Due to the target ship in the traditional ship handling simulator have not the ability to give way to other ships automatically to avoid collision, this paper put forward a new idea that bringing the hydraulic servo platform, six degrees of freedom ship mathematical model, the actual traffic flow, researching achievement of automatic anti-collision in research of the new pattern ship handling simulator, and successfully develop the Intelligent Ship Handling Simulator(ISHS for short). The paper focuse on the research on the network communication model of ISHS. We took the entire simulator system as three relatively independent networks, proposed a framework of communication network that combined IOCP model based on TCP with blocking model based on UDP, and gave the communication process and protocols of system. Test results indicate that this is an effective way to improve the ownship capacity of ship handling simulator and meet the need of multi-ownship configuration of desktop system of ship handling simulator.

Download Full-text

A Reinforcement Learning Approach for Interference Management in Heterogeneous Wireless Networks

International Journal of Interactive Mobile Technologies (iJIM) ◽

10.3991/ijim.v15i12.20751 ◽

2021 ◽

Vol 15 (12) ◽

pp. 65

Author(s):

Akindele Segun Afolabi ◽

Shehu Ahmed ◽

Olubunmi Adewale Akinola

Keyword(s):

Reinforcement Learning ◽

Power Level ◽

Heterogeneous Wireless Networks ◽

Interference Management ◽

Base Station ◽

User Equipment ◽

Base Stations ◽

Multi Agent Systems ◽

Q Learning ◽

Macro Cell

<span lang="EN-US">Due to the increased demand for scarce wireless bandwidth, it has become insufficient to serve the network user equipment using macrocell base stations only. Network densification through the addition of low power nodes (picocell) to conventional high power nodes addresses the bandwidth dearth issue, but unfortunately introduces unwanted interference into the network which causes a reduction in throughput. This paper developed a reinforcement learning model that assisted in coordinating interference in a heterogeneous network comprising macro-cell and pico-cell base stations. The learning mechanism was derived based on Q-learning, which consisted of agent, state, action, and reward. The base station was modeled as the agent, while the state represented the condition of the user equipment in terms of Signal to Interference Plus Noise Ratio. The action was represented by the transmission power level and the reward was given in terms of throughput. Simulation results showed that the proposed Q-learning scheme improved the performances of average user equipment throughput in the network. In particular, </span><span lang="EN-US">multi-agent systems with a normal learning rate increased the throughput of associated user equipment by a whooping 212.5% compared to a macrocell-only scheme.</span>

Download Full-text

Path Planning in Complex Environments for Industrial Robots with Additional Degrees of Freedom

Romansy 13 - CISM International Centre for Mechanical Sciences ◽

10.1007/978-3-7091-2498-7_46 ◽

2000 ◽

pp. 431-438 ◽

Cited By ~ 2

Author(s):

Francisco Valero ◽

Vicente Mata ◽

Marco Ceccarelli

Keyword(s):

Path Planning ◽

Degrees Of Freedom ◽

Industrial Robots ◽

Complex Environments

Download Full-text

Degrees-Of-Freedom in Multi-Cloud Based Sectored Cellular Networks

Entropy ◽

10.3390/e22060668 ◽

2020 ◽

Vol 22 (6) ◽

pp. 668

Author(s):

Samet Gelincik ◽

Ghaya Rekaya-Ben Othman

Keyword(s):

Cellular Networks ◽

Degrees Of Freedom ◽

Signal To Noise Ratio ◽

Base Stations ◽

Processing Capacity ◽

Coding Schemes ◽

Minimum Number ◽

Cut Set ◽

Base Band ◽

Multi Cloud

This paper investigates the achievable per-user degrees-of-freedom (DoF) in multi-cloud based sectored hexagonal cellular networks (M-CRAN) at uplink. The network consists of N base stations (BS) and K ≤ N base band unit pools (BBUP), which function as independent cloud centers. The communication between BSs and BBUPs occurs by means of finite-capacity fronthaul links of capacities C F = μ F · 1 2 log ( 1 + P ) with P denoting transmit power. In the system model, BBUPs have limited processing capacity C BBU = μ BBU · 1 2 log ( 1 + P ) . We propose two different achievability schemes based on dividing the network into non-interfering parallelogram and hexagonal clusters, respectively. The minimum number of users in a cluster is determined by the ratio of BBUPs to BSs, r = K / N . Both of the parallelogram and hexagonal schemes are based on practically implementable beamforming and adapt the way of forming clusters to the sectorization of the cells. Proposed coding schemes improve the sum-rate over naive approaches that ignore cell sectorization, both at finite signal-to-noise ratio (SNR) and in the high-SNR limit. We derive a lower bound on per-user DoF which is a function of μ BBU , μ F , and r. We show that cut-set bound are attained for several cases, the achievability gap between lower and cut-set bounds decreases with the inverse of BBUP-BS ratio 1 r for μ F ≤ 2 M irrespective of μ BBU , and that per-user DoF achieved through hexagonal clustering can not exceed the per-user DoF of parallelogram clustering for any value of μ BBU and r as long as μ F ≤ 2 M . Since the achievability gap decreases with inverse of the BBUP-BS ratio for small and moderate fronthaul capacities, the cut-set bound is almost achieved even for small cluster sizes for this range of fronthaul capacities. For higher fronthaul capacities, the achievability gap is not always tight but decreases with processing capacity. However, the cut-set bound, e.g., at 5 M 6 , can be achieved with a moderate clustering size.

Download Full-text

A Security Scheme of 5G Ultradense Network Based on the Implicit Certificate

Wireless Communications and Mobile Computing ◽

10.1155/2018/8562904 ◽

2018 ◽

Vol 2018 ◽

pp. 1-11 ◽

Cited By ~ 3

Author(s):

Zhonglin Chen ◽

Shanzhi Chen ◽

Hui Xu ◽

Bo Hu

Keyword(s):

Data Communication ◽

Base Stations ◽

System Capacity ◽

Communication Model ◽

Spatial Reuse ◽

Network Bandwidth ◽

Security Problem ◽

Small Base ◽

And Performance ◽

Security Communication

The ultradense network (UDN) is one of the most promising technologies in the fifth generation (5G) to address the network system capacity issue. It can enhance spatial reuse through the flexible, intensive deployment of small base stations. A universal 5G UDN architecture is necessary to realize the autonomous and dynamic deployment of small base stations. However, the security of the 5G UDN is still in its infancy, and the data communication security among the network entities is facing new challenges. In this paper, we proposed a new security based on implicit certificate (IC) scheme; the scheme solves the security problem among the access points (APs) in a dynamic APs group (APG) and between the AP and user equipment (UE). We present each phase regarding how two network entities obtain the Elliptic Curve Qu-Vanstone (ECQV) implicit certificate scheme, verify each other’s identity, and share keys in an UDN. Finally, we extensively analyze our lightweight security communication model in terms of security and performance. The simulation on network bandwidth evaluation is also conducted to prove the efficiency of the solution.

Download Full-text

Research on Pulse Jet Action Angle Optimization

Applied Mechanics and Materials ◽

10.4028/www.scientific.net/amm.401-403.1628 ◽

2013 ◽

Vol 401-403 ◽

pp. 1628-1631

Author(s):

Zai Ke Tian ◽

Suo Chang Yang ◽

De Long Feng ◽

Yun Zhi Yao

Keyword(s):

Control Strategy ◽

Degrees Of Freedom ◽

Impact Point ◽

Force Action ◽

Optimization Strategy ◽

Pulse Force ◽

Trajectory Simulation ◽

Trajectory Correction ◽

6 Degrees Of Freedom ◽

Pulse Jet

To determine the action angle of is the important topics on Trajectory Correction technology.The pulse force action angle and residual impact point deviation were theoretically analyzed on the basis of traditional control strategy of pulse jets. It has been found that there may be a large residual impact point deviation when the correction ability is different. An optimization strategy for the pulse force action angle control was presented, and the method was verified by the 6-degrees of freedom trajectory simulation.

Download Full-text