Objective To establish a mathematical model of tumor growth and invasion under radiotherapy, so as to numerically simulate the effect of radiotherapy on tumor growth and make sensitivity analysis.Methods The mathematical model of tumor growth and invasion with time evolution before and after radiotherapy was established. The model included four key variables in the process of tumor invasion: tumor cells, extracellular matrix （ECM）, matrix-degradative enzymes （MDEs） and oxygen. The linear quadratic (LQ) model was used to simulate the survival probability of tumor cells after radiotherapy, and the effects of different radiotherapy schemes and radiotherapy coefficients on the treatment effect were discussed. Traditional radiotherapy and intraoperative targeted radiotherapy were compared.Results Under the premise of constant total dose, the results of radiotherapy were directly proportional to the radiotherapy coefficient, but not related to the radiotherapy frequency; the therapeutic effect of intraoperative targeted radiotherapy was better than that of standard treatment.Conclusions Simulation results are basically consistent with clinical experimental results. As a more efficient treatment method, intraoperative targeted radiotherapy can provide new ideas for clinical tumor treatment.