Abstract: Fish is one of the most biodiversity species on the economic benefits and production value of fisheries. Fish diseases have posed major risks to the development of fisheries at the farm level. Fortunately, quality screening can be carried out to keep healthy and high-quality fish species, when fish stocks are in the fry stage. The diseased and low-quality fish species can then be removed. But the manual work is very labor-intensive. Machine vision techniques can be utilized to effectively screen diseased fish. However, it is still lacking in the obvious disease characteristics, particularly on the small target. In this study, an improved Yolov8 model was proposed to detect fish diseases using multi-label compensation. Firstly, a device was designed to separate fish schools in advance, which was used to separate the diseased fish. A Koi dataset with injuries and diseases was constructed to simulate the way individual fish pass through the screening device. Efficient deployment of the model was realized in the device. According to the pixel size occupied by easily observable diseases on the fish body, the severity of the disease was classified into three categories: simple, moderate, and complex. Finally, a dataset sample was obtained to contain 5 920 images, which was divided into a training set, validation set, and test set in an 8:1:1 ratio. Then, the MS-YOLOv8 network was designed using the YOLOv8n model. Among them, the SPD convolution was used to replace the cross-row convolution operation in key parts of the original network, in order to reduce the loss of feature information during image downsampling. A multi-label loss function was proposed to balance the classification loss and bounding box loss when training small target diseases in the network. The Intersection over Union (IoU) between a single prediction box and multiple labels was calculated to expand its receptive field when reducing loss values. More contextual information was utilized from the target. There was a better balance for the decrease in classification caused by IoU loss optimization in the network, thereby reducing the missed detection rate of diseases. The experimental results show that when YOLOv8 uses MLIoU, the detection rate for fish diseases in easy, normal, and difficult is 11.13, 3.76, and 12.38 percentage points higher than when using the latest method Inner-SIoU. Compared with the original model, it was improved by 6.27, 0.66, and 3.01 percentage points. Furthermore, MS-YOLOv8 mAP values were 12.05, 10.18, and 11.15 percentage points higher than those of YOLOv5n, YOLOv7, and YOLOv8n, respectively. The total detection rate was 95.36%, and the image detection time was 132 frames per second, indicating the high comprehensive performance. In the subjective test, MS-YOLOv8 was used to suppress background disturbance similar to fish disease. Many kinds of fish were detected, such as grass carp and silver carp. Excellent generalization and robustness were found in the binary classification of the MS-YOLOv8 model. The probability of screening diseased fish was improved, especially when the target of the disease was small, but the location of the fish was concentrated. The finding can also provide effective technical support to the clean-up of diseased fish in fishery.