Abstract: Terraced fields, as an important arable land resource in mountainous areas, have various ecosystem service functions and values. The changes in their quantity are closely related to national food security and sustainable land use. However, with the increasing contradictions between people, land, environment, and resources, the phenomenon of terraced fields being abandoned is becoming increasingly severe. Therefore, analyzing its spatiotemporal evolution characteristics and driving factors is of great significance for ensuring food security and promoting the inheritance and development of terraced agricultural cultural heritage. This study takes the five major rice terraces in southern China as the research object. Based on remote sensing image data from 2010 to 2020, the distribution data of terraces in each region was visually interpreted and verified. Firstly, calculation methods such as kernel density and landscape pattern index were used to reveal the spatiotemporal distribution changes of terraces in different regions from the characteristics of terrace area changes, kernel density analysis of terrace patch changes, and terrace patch pattern changes. On this basis, 11 indicators were selected as driving factors from both natural and socio-economic aspects, and a geographic detector model was used to explore the driving factors of spatiotemporal changes in terraces in different regions. Based on this, relevant feasible measures were proposed. The results showed that: 1) From 2010 to 2020, there were varying degrees of abandonment of terraced fields in each region, mainly in areas with fragmented edges. Among them, the abandonment of terraced fields in Ziquejie was the highest (25.61%), and the abandonment of terraced fields in Jiabang was the lowest (4.75%). In comparison, the abandonment of terraced fields in Ziquejie and Union Terraced Fields decreased; 2) More than 80% of the terraced fields are concentrated in medium to high and high-density areas, and the spatial distribution of terraced patches varies significantly in different regions. Moreover, due to the impact of abandoned cultivation, the range of high-density areas has narrowed; 3) The calculation results of landscape pattern index show that the fragmentation degree of terraced fields in Ziquejie is the highest, followed by Jiabang, Longji, and Union Terraced Fields. The fragmentation degree of terraced patches in the core area of Hani Terraced Fields is the lowest, and from 2010 to 2020, the fragmentation degree of terraced patches in Jiabang, Longji, and Union Terraced Fields showed a decreasing trend, while the fragmentation degree of terraced patches in Ziquejie and Hani Terraced Fields core areas showed an increasing trend; 4) The factor detection results showed that the strongest effects on the terraced fields in the core areas of Jiabang, Longji, Ziquejie, Union, and Hani terraced fields were the distance from the highway, distance from residential areas, slope, terrain undulation, and altitude, respectively. Meanwhile, the interaction results showed that the influence of the interaction between terrain and accessibility conditions significantly increased, indicating that terrain and accessibility conditions play an important role in the spatiotemporal distribution changes of terraced fields. The research results can provide scientific basis and theoretical reference for the protection and sustainable development of terraced field resources in mountainous areas of China.