The inspection of defects is one of the most important aspects in the quality inspection of raw silk. We introduce a raw-silk defect detection system based on image vision and image analysis that is accurate and objective. In the experimental phase, we develop an image-acquisition section—which includes a charge-coupled device (CCD) image sensor, a telecentric lens, a light source, and a raw-silk winding device to capture the raw silk images steadily. After the image capture stage, an image-processing section tasked with threshold segmentation and morphology operations is carried out to obtain the defects of raw silk. To classify the raw-silk defects accurately and quickly, we propose an area method for the classification of raw-silk defects into five categories: larger defects, large defects, common defects, small defects, and smaller defects. Meanwhile, in order to recognize the common raw-silk defects—e.g., Bavella silk, nodes, and loose ends—that cannot be detected by the Uster evenness tester, the moment invariants of each segmented region of the images are extracted and used as the input of support vector machine(SVM).A SVM is designed as a classifier to recognize the samples. The experimental results show that the proposed method can recognize these common raw-silk defects effectively. According to the new classification and accurate recognition of raw-silk defects using the proposed method, we can improve the inspection standards for raw silk and advise raw-silk reeling enterprises seeking to optimize the technological parameters.