| In this thesis,we investigate optimal control problems of Marko-vian and non-Markovian stochastic systems described by coupled forward-backward stochastic differential equations(FBSDE),solve the optimal con-trols when the performance functions under g-expectation obtain the maxi-mum value,derive and prove the stochastic maximum principles.The the-oretical results are used to solve two optimal control problems in finance and insurance.We give a forward stochastic differential equation driven by a con-trolled It(?)-L(?)vy process firstly.Then,by the definition of g-expectation,we construct a backward stochastic differential equation,transform the origi-nal problem into the optimal control problem of a coupled FBSDE,and give the performance function under g-expectation.When the stochastic system is Markovian,the maximum principle is deduced and proved under the assumption that the Hamiltonian is concave.In the study of the optimal control problem of a non-Marovian stochastic system,the drift and diffu-sion coefficients in the system and the functions in the performance function are assumed to be stochastic.We introduce Malliavin calculus to define the modified adjoint processes and corresponding functions,expressed in terms of Malliavin derivatives,derive and prove the maximum principle for the non-Marovian system.Two practical problems have been solved by using Malliavin calcu-lus and g-expectation in the end of this thesis.The front part is solving the mean-variance portfolio selection problem of stochastic system driven by It(?)-L(?)vy process.The expressions of optimal terminal wealth and opti-mal portfolio are solved by Clark-Ocone formula,and the optimal control expressions are given for wealth instances.An optimal dividend problem driven by It(?)-L(?)vy process in insurance market is solved in the later part,and its conditions for the optimal dividend rate are given. |