Spectrum sensing is of the utmost importance to the workings of a cognitive radio network (CRN). The spectrum has to be sensed to decide whether the cognitive radio (CR) user can transmit or not. Transmitting on unoccupied spectrum becomes a hard task if energy-constrained networks are considered. CRNs are ad hoc networks, and thus, they are energy-limited, but energy harvesting can ensure that enough energy is available for transmission, thus enabling the CRN to have a theoretically infinite lifetime. The residual energy, along with the sensing decision, determines the action in the current time slot. The transmission decision has to be grounded on the sensing outcome, and thus, a combined sensing–transmission framework for the CRN has to be considered. The sensing–transmission framework forms a Markov decision process (MDP), and solving the MDP problem exhaustively through conventional methods cannot be a plausible solution for ad hoc networks such as a CRN. In this paper, to solve the MDP problem, an actor–critic-algorithm-based solution for optimizing the action taken in a sensing–transmission framework is proposed. The proposed scheme solves an optimization problem on the basis of the actor–critic algorithm, and the action that brings the highest reward is selected. The optimal policy is determined by updating the optimization problem parameters. The reward is calculated by the critic component through interaction with the environment, and the value function for each state is updated, which then updates the policy function. Simulation results show that the proposed scheme closely follows the exhaustive search scheme and outperforms a myopic scheme in terms of average throughput achieved.
Routing for network capacity maximization in energy-constrained ad-hoc networks