Abstract: Shrimp has been one of the most favorite seafood for years. It is essential to timely estimate the amount of feed left in the bait tray after feeding in shrimp aquaculture. Feeding strategies can also be adjusted to reduce the bait costs in recent years. The traditional detection of residual baits can rely on visual inspection by the shrimp farmers. Neural networks and deep learning have been introduced to detect and count the residual baits at present. However, large-scale neural networks cannot be successfully implemented on mobile devices, due mainly to the low recognition accuracy and large model computation. In this study, the improved model was proposed to estimate the density map of residual baits using a hybrid dilated convolution and attention multi-scale network (HAMNet). The high accuracy and low complexity were achieved in the detection models of residual bait. The HAMNet model was divided into three components: A low-level feature extractor (LLFE), a high-level feature extractor (HLFE), and a density map restorer and generator (DMRG). These components served as the front-end, the middle-end, and the back-end network of the improved model, respectively. Firstly, inspired by the multi-column convolutional neural network (MCNN), the parallel convolution block (PCB) was designed in the front-end network, in order to extract the feature information of residual bait at multiple scales within a single-column architecture; At the same time, the hybrid dilated convolution block (HDCB) was introduced into the mid-end network to expand the receptive field, in order to further learn the multi-scale features. Secondly, a channel attention mechanism (CAM) was embedded into the network to recalibrate the weights of useful feature information using the interdependence among channels, in order to highlight the difference between the target and background. Finally, the learnable transposed convolutional layers were applied in the back-end network to recover the detailed information from the feature maps. The quality of density maps was improved to reduce the counting errors. As such, the high-quality density map was then obtained during downsampling in the front-end network. In addition, the effectiveness of the improved model was validated using residual bait images under bait tray conditions. A comparison was also implemented with the classical networks of density map estimation. Comparative experiments showed that the HAMNet model was achieved in the minimum mean absolute error (MAE) of 2.0, the minimum root mean square error (RMSE) of 2.9, and the least floating point operations (FLOPs) at 6.55 G on the residual bait datasets, with a parameter count of only 0.52MB. The HAMNet model shared the higher counting accuracy and stability with the lower computational complexity. Compared with the baseline MCNN network, the improved model achieved a 44.4% reduction in MAE, a 40.8% reduction in RMSE, and a 13.7% reduction in FLOPs. Compared with the CMTL, SANet, and CSRNet, the optimal balance was obtained in all performance metrics. In summary, the HAMNet model outperformed the rest, in terms of overall performance, thus improving the counting accuracy with the low computational volume. The finding can provide a strong reference to rapidly quantify the residual baits in shrimp aquaculture. Novel ideas were also offered to deploy the detection models of residual bait on the platforms with limited computational power.