Abstract: Clay cutting is of great significance in the optimal structure and motion parameters of cutting tools in agricultural machinery. The better performance of soil tillage can be achieved to reduce the work power consumption, and cutting resistance. However, clay shear failure has posed a serious challenge to agricultural cultivation. Simulation and prediction of shear resistance are also difficult so far, due to the mechanical properties and the complex deformation of clay during shear failure. In this study, a clay shear failure model was established to accurately describe the shear failure process of clay, and then estimate the shear force using a smoothed-particle hydrodynamics (SPH) framework. A Grady-Kipp tensile damage failure model was constructed using the elastic-plastic constitutive model. A mixed failure model was obtained to couple the two models. The kernel function was chosen as a cubic spline function under the SPH framework, with a smooth length 1.3 times the particle spacing. The tensile instability was also reduced using numerical techniques, such as artificial viscosity, artificial stress, and XSPH. After that, the contact algorithm was used to calculate the cutting resistance between the tool and the clay during cutting using the Fortran language on the Visual Studio 2019 platform. The velocity, displacement, and stress distribution maps were then captured using Tecplot software. The parameters of the model were obtained via a series of geotechnical tests. Specifically, the direct shear tests were to obtain the cohesion and internal friction angle, while the ring knife method was to measure the density of clay. The cutting experiments were conducted to validate the model using boxes with acrylic panels. The results show that there was better consistency between the experiment and simulation, in terms of cutting failure deformation. The development direction of cracks was also aligned better with the shape of the cracks. The evolution of tool-cutting resistance was predicted during clay-cutting failure, according to the previous clay-tool interaction model. The cutting resistance fluctuated in the early stage, and then stabilized near a certain value in the later stage, as the displacement increased. The reason was that there was a gap between clay particles at the beginning of cutting, leading to initial instability. As the cutting process proceeded, the particles were rearranged into a more compact state, indicating a gradually stable curve. A systematic analysis was also implemented to determine the influence of cutting depth on the failure morphology and cutting resistance during clay cutting. It was found that the larger the cutting depth was, the larger the blocks formed during clay crushing, and the greater the cutting resistance was. At the same time, there was also some influence of the tool friction coefficient on the cutting resistance. The smallest cutting resistance was then achieved when the friction coefficient was 0.1. Therefore, the mixed failure model can be expected to better accurately describe the clay cutting failure, the propagation of cracks, as well as the formation of clay fragments and granular particles during clay cutting, compared with the shear. This finding can provide a strong reference for the numerical simulation of clay cutting.