Abstract: To obtain the optimal capacity and analyze the relationship between the optimal Levelized Cost of energy and the loss of power supply probability of a wind/photovoltaic/battery hybrid power generation system, a capacity optimization method based on an improved artificial bee colony algorithm was proposed. Firstly, the component models of the hybrid power generation system (wind turbine, photovoltaic, battery, and grid loads) were established and the rule-based energy management strategy was carefully designed to coordinate the different components of the hybrid power generation system. According to whether the electric power generated and converter capacity was sufficient, the energy management strategy was designed as four scenarios and the corresponding rules were designed; Secondly, the model for the capacity optimization problem was built with the objective function of minimizing the Levelized Cost of energy, where the requirement of satisfying the loss of power supply probability was handled as a constraint; Lastly, the penalty function method was applied to handle the loss of power supply probability constraint and an improved artificial bee colony was applied to improve the accuracy of the solution to the capacity optimization problem. In the algorithm, differential evolution operators were introduced to balance the bees’ ability of exploration and exploitation in different stages of the optimization process and a new food source could be generated by differential evolution operators with a probability increasing adaptively with the iterations in the employed bee stage. To verify the effectiveness of the algorithm, contrast tests with artificial bee colony and differential evolution were performed. The comparison results showed the proposed algorithm had better accuracy, robustness, and convergence rate. Then, to obtain the relationship between the optimal Levelized Cost of energy and the loss of power supply probability, capacity optimizations of the hybrid system was performed under the different loss of power supply probability requirements, which was changed from 0 to 5% with an equal step of 0.1%. Results showed that the Levelized Cost of energy decreases with the increase of loss of power supply probability and the descent rate was gradually decreased. This implied that it could obtain more obvious economic benefits by reducing the reliability requirement appropriately when the loss of power supply probability was at a smaller value. Furthermore, the influence of main parameters, such as the initial capital of the main component (wind turbine, photovoltaic, and battery) and meteorological resources (annual average of solar radiation and wind speed) on the Levelized Cost of energy was carried out by sensitivity analyses. The sensitivity analyses of the initial capital of the components were carried out by capacity optimizations of the hybrid system with different initial capital values of (wind turbine, photovoltaic, and battery system), which were changed from 60% to 140% of their best estimate value, the results showed that the influence of initial capital on the Levelized Cost of energy presented a linear relationship, the line of the photovoltaic was with the highest slope, and the line of the battery had the lowest slope. This implied that it could generate more obvious economic benefits by reducing the photovoltaics’ initial capital. The sensitivity analyses of the meteorological resources were carried out by capacity optimizations of the hybrid system with different annual average values of solar radiation and wind speed, which were changed from 80% to 120% of their best estimate value, the results showed that the influence of meteorological resources on the Levelized Cost of energy presented a linear relationship, and the slope of wind speed was higher than that of the solar radiation. This implied that it should pay more attention to the wind resource evaluation.