Rolling mill system may lose its stability due to the change of lubrication conditions. Based on the rolling mill vertical–torsional–horizontal coupled dynamic model with nonlinear friction considered, the system stability domain is analyzed by Hopf bifurcation algebraic criterion. Subsequently, the Hopf bifurcation types at different bifurcation points are judged. In order to restrain the instability oscillation induced by the system Hopf bifurcation, a linear and nonlinear feedback controller is constructed, in which the uncoiling speed of the uncoiler is selected as the control variable, and variations of tensions at entry and exit as well as system vibration responses are chosen as feedback variables. On this basis, the linear control of the controller is studied using the Hopf bifurcation algebraic criterion. And the nonlinear control of the controller is studied according to the center manifold theorem and the normal form theory. The results show that the system stability domain can be expanded by reducing the linear gain coefficient. Through choosing an appropriate nonlinear gain coefficient, the occurring of the system subcritical bifurcation can be suppressed. And system vibration amplitudes reduce as the increase of the nonlinear gain coefficient. Therefore, introducing the linear and nonlinear feedback controller into the system can improve system dynamic characteristics significantly. The production efficiency and the product quality can be guaranteed as well.