Training an Automated Directional Drilling Agent with Deep Reinforcement Learning in a Simulated Environment

Drilling a directional well becomes an essential process in the oil and gas industry to ensure better reservoir exposure and less wellbore collision risk. In the high-volume drilling market, cost-effective mud motors are dominant. The motor is capable of delivering the desired well curvature by switching between rotating and sliding operations. Therefore, to follow a predefined well trajectory, it is a critical mission to determine the optimal operation control sequence of the motor. In this paper, a method of training an automatic agent for motor directional drilling using the deep reinforcement learning approach is proposed. In designing the method, motor-based directional drilling is framed into the reinforcement learning with an automatic drilling system, also known as an agent, interacting with an environment (i.e., formations, wellbore geometry, equipment) through choices of controls in a sequence. The agent perceives the states such as inclination, MD, TVD at survey points and the planned trajectories from the environment, and then decides the best action of sliding or rotating to achieve the maximum total rewards. The environment is affected by the agent's actions and returns corresponding rewards to the agent. The rewards can be positive (such as drilling to target) or negative (such as offset distance to the planned trajectory, cost of drilling, and action switching). To train our agent, currently, a drilling simulator in a simulated environment is created with layered earth model and BHA directional responses in layers. Other attributes of the drilling system are assumed to be constant and handled automatically by the simulator. The planned trajectory is also provided to the agent while training. The directional-drilling agent is trained for thousands of episodes. As a result, the agent can successfully drill to target in this simulated environment through the decisions of sliding and rotating. The proposed workflow is known as the first automated directional drilling method based on deep reinforcement learning, which makes a sequence of decisions of rotating and sliding actions to follow a planned trajectory.