Abstract: Utilizing the convolutional neural network (CNN) algorithm for grading mango appearance quality could achieve high accuracy. Due to the CNN algorithm is computation-intensive and storage-intensive, it is challenging to implement CNN on low-cost and performance-constrained edge devices, which greatly limits the application and promotion of CNN based methods for mango appearance quality grading. To address this problem, this paper takes into account the grading accuracy, speed, power consumption, and cost of mango appearance quality grading detection, and proposes a grading device based on FPGA-accelerated CNN. Firstly, a lightweight network called “Compact MobileNet” (CMNet) with a simple and efficient structure was designed. The parameter and computational complexity of CMNet were significantly reduced by compressing the network structure, making it suitable for deployment on edge devices while ensuring an acceptable accuracy. Secondly, in order to accelerate the execution speed of CMNet on cost- and performance- constrained edge devices, BN layer fusion and model quantization methods were used to further reduce the storage requirements and computational complexity of the model. Meanwhile, an FPGA-based hardware accelerator for CMNet was designed. As the basic function of the accelerator achieved, the high-level synthesis (HLS) optimization methods, including “unrolling the for-loop”, “pipelining the for-loop”, and “array partitioning”, were used to optimize the parallelism of the hardware accelerator. Finally, a device for grading the appearance quality of mangoes was developed using an FPGA development board, specifically the “ZYNQ Z7 Lite 7020” model. This device integrates an OV5640 camera module, the CMNet network acceleration circuit, and an HDMI interface, enabling mango image collection, real-time appearance quality detection, and the display of results. To verify the performance of CMNet and the grading device, experiments were conducted in a laboratory environment. The Dosehri mango dataset, downloaded from internet, was used to train and validate CMNet. The mango appearance quality was classified into three grades according to China's agricultural industry standard for mango (NY/T 492-2002). The model comparison experiment demonstrates that CMNet has very lightweight model scale, only with 0.015M parameters and 7.2MFlops computations, while maintaining a high grading accuracy of 96.4%. This indicates that CMNet is suitable for deployment on a resource-limited FPGA terminal. The assessment experiments of the FPGA-based accelerator of CMNet demonstrate that the accelerator can significantly expedite the quality grading process, achieving a speed of 0.11 s per frame, this indicates that the accelerator is capable of detecting the mango appearance quality in real-time. The performance comparison experiment among various computing devices reveals that the power consumption of the FPGA-based grading device is 2.6 W, the lowest among the tested devices. In comparison to CPUs and GPUs, the FPGA-based detection device offers substantial advantages in terms of power consumption and portability. When compared to portable Raspberry Pi devices and Android smartphones, the FPGA excels in the best unit cost performance. In summary, all experimental results indicate that this paper presents a low-cost, low-power, high-accuracy, and high-speed grading device for assessing mango appearance quality. This device can be effectively utilized for real-time grading of mango quality in the field. Furthermore, this paper provides a reference solution for similar agricultural applications where artificial intelligence methods are employed to rapidly extract information from video data.