Abstract:A large mode area multi-core orbital angular momentum (OAM) transmission fiber is designed and optimized by neural network and optimization algorithms. The neural network model has been established first to predict the optical properties of multi-core OAM transmission fibers with high accuracy and speed, including mode area, nonlinear coefficient, purity, dispersion, and effective index difference. Then the trained neural network model is combined with different particle swarm optimization algorithms for automatic iterative optimization of multi-core structures respectively. Due to the struc-tural advantages of multi-core fiber and the automatic optimization process, we designed a number of multi-core struc-tures with high OAM mode purity (>95%) and ultra-large mode area (>3000μm2), which is larger by more than an order of magnitude compared to the conventional ring-core OAM transmission fibers.