This study takes the high school mathematics classroom as the research object. By systematically analyzing the problems of teachers’ teaching behaviors, it explores the classroom teaching optimization strategies based on the analysis of teaching behaviors. Research has found that current high school mathematics classrooms have problems such as monotonous teaching behaviors, insufficient interaction between teachers and students, and a lack of hierarchical problem design. In response to these problems, optimization strategies such as optimizing teaching behavior design, strengthening classroom interaction mechanisms, improving problem-driven strategies, and innovating teaching evaluation methods have been proposed. Research shows that through scientific analysis and optimization of teaching behaviors, the quality of high school mathematics classroom teaching can be effectively improved, and the development of students’ core mathematical literacy can be promoted.