| The origin of quark and lepton masses and flavor mixing is a longstanding open question in particle physics,also known as the flavor puzzle.In recent years,a method based on modular symmetry is an important advance in this field.It relates the Yukawa couplings to the modular forms in mathematics which is a class of highly nontrivial holomorphic functions,thus making the theory highly predictive.In this paper,we develop this theoretical approach from some aspects.After introducing some necessary background,we first present two module-invariant fermion mass models based on A4 and S4 in the original modular invariant framework,which successfully explain the experimental data for the currently known flavor parameters with a small number of input parameters.Then we extend the original modular invariance approach to include the integral weight and rational weight modular forms,meanwhile,the finite flavor group is extended from the inhomogeneous finite modular group ΓN to the homogeneous finite modular group Γ’N and the metaplectic finite modular groupΓN.This greatly enriches the possibilities for building new flavor models.In addition,we extend the original modular invariant theory to the general higher-dimensional moduli space G/K,and the modular symmetry is extended from SL(2,Z)to the general discrete subgroup Γ of G,and the modular invariance requires the Yukawa couplings to be the automorphic forms.We take G=Sp(2g,R),K=U(g),Γ=Sp(2g,Z)as an example to establish the so-called Sp(2g,Z)symplectic modular invariant supersymmetry theory,it is the most natural high-dimensional generalization of the SL(2,Z)modular invariant theory.We construct several viable lepton and quark mass models based on the genus g=2.Numerical analysis shows that the existence of multi-moduli fields is helpful to solve the mass hierarchy problems.In order to further enhance the predictive power of the theory,we combine symplectic modular symmetry with generalized CP symmetry self-consistently,and show that for genus g≥3 the definition of CP is unique,while two independent possibilities are allowed when g≤2.We give the CP transformation properties of the matter fields,moduli fields and modular form multiplets,and construct a symplectic modular invariant lepton model with generalized CP,in which the predicted Dirac CP violation phase,the lepton mixing angles and the lepton masses are all derived from a nontrivial moduli vacuum.Finally,we revisit the original modular-invariant framework from the point of view of vector-valued modular form.The Yukawa couplings are represented by the more general vector modular forms,and the finite flavor group is extended from ΓN,Γ’N to the generalized finite modular group Gf=Γ/G,which is the quotient group of the Γ and its normal subgroup G.This allows modular symmetry to cover a wide range of flavor symmetries,and provides a systematic differential equation method for constructing all possible modular form multiplies.Moreover,we present in the appendix quite a number of theoretical tools that are necessary in the construction of relevant modular-invariant masses models.These works greatly extend the framework of modular invariant supersymmetry theory,provide a large number of tools for model building,and indicate a broad prospect of applying modular symmetry to solve the flavor puzzle. |