Stochastic Optimal Control Problems With Markov Chains And Applications To Finance

This thesis is concerned with stochastic optimal control problems with Markov chains and their applications to fmance.On the one hand,we develop the theory of stochastic optimal control problems with Markov chains,including the stochastic max-imum principle,dynamic programming principle,and the relationship between them.On the other hand,we apply the theoretical results to solve some problems in finan-cial mathematics,such as stock trading,portfolio selection,utility maximization,and investment-consumption,etc.In practice,many phenomena and systems have the properties of regime-switching or trend-changing.Normally,these cases can be described by a continuous-time,finite-state Markov chain.For example,a stock market can be regarded as a bull market or a bear market at a given time.In bull market,the expected return rate of the stock is positive and in bear market,the expected return rate of the stock is negative.Moreover,the volatility of bull market is usually smaller than that of bear market.We use a two-state Markov chain to represent the trend of the market,one state stands for bull market and the other stands for bear market.In this way,we are able to make the stock price equation depend on the trend of the market.In view of this example,we see that the Markov chain well reflects the random changes of the environment,and it is meaningful to study the stochastic optimal control problems with Markov chains.Compared with the classical pure diffusion models,regime switching models have two advantages.First,the discrete jump Markov process captures more directly the dynamics of events that are less frequent(occational)but nevertheless more significant to long-term system behavior.For example,the Markov chain can represent discrete events such as market trends and other economic factors that are difficult to be incorporated into a diffusion model.Second,on the aspect of numerical computation,regime-switching models are also very convenient.Not only the corresponding Hamilton-Jacobi-Bellman(HJB)equations are very simple,but also they require very limited data input,for example,the return rate b(i),the volatility ?(i)for each state i,and the Q matrix.Consequently,the stochastic optimal control problems with Markov chains have the advantages on both theory and numerics.Now we give the main contents and organization of this thesis as follows.Chapter 1 is concerned with the optimal switching problem under a hybrid diffusion(or,regime switching)model in an infinite horizon.The state of the system consists of a number of diffusions coupled by a finite-state continuous-time Markov chain.Based on the dynamic programming principle,the value function of our optimal switching prob-lem is proved to be the unique viscosity solution to the associated system of variational inequalities.The optimal switching strategy,indicating when and where it is optimal to switch,is given in terms of the switching and continuation regions.In many applica-tions,the underlying Markov chain has a large state space and exhibits two-time-scale structure.In this case,a singular perturbation approach is employed to reduce the com-putational complexity involved.It is shown that as the time-scale parameter ? goes to zero,the value function of the original problem converges to that of a limit problem.The limit problem is much easier to solve,and its optimal switching solution leads to an approximate solution to the original problem.Finally,as an application of our theo-retical results,an example concerning the stock trading problem in a regime switching market is provided.Both the optimal trading rule and convergence result are numerically demonstrated in this example.In Chapter 2,we consider a continuous-time mean-variance portfolio selection prob-lem with random time horizon in an incomplete market.We deal with this problem using results of stochastic linear-quadratic(LQ)optimal controls and backward stochastic dif-ferential equations(BSDEs).Specifically,the mean-variance problem will be formulated as a linearly constrained stochastic LQ optimal control problem.Moreover,the solv-ability of this LQ problem will be reduced to the global solvability of two BSDEs.One is conventionally called a stochastic Riccati equation,and the other is referred to as an auxiliary BSDE.We apply the theory of martingales of bounded mean oscillation,briefly called BMO-martingales,to provide a direct and greatly simplified proof of the solvabil-ity of the two BSDEs.We shall derive closed-form expressions for both the optimal portfolios and the efficient frontier in terms of the solutions of the two BSDEs.In Chapter 3,we prove a sufficient stochastic maximum principle for the optimal control of a forward-backward Markov regime switching jump diffusion system and show its connection to dynamic programming principle.The result is applied to a cash flow valuation problem with terminal wealth constraint in a financial market.An explicit optimal strategy is obtained in this example.Chapter 4 is concerned with a Pontryagin's maximum principle for optimal control problem of stochastic system,which is described by an anticipated forward-backward stochastic differential delayed equation and modulated by a continuous-time finite-state Markov chain.We establish a.necessary maximum principle and a sufficient verification theorem for the optimal control by virtue of the duality method and the convex analysis.To illustrate the theoretical results,we apply them to a recursive utility investment-consumption problem and the optimal consumption rate is derived explicitly.Then we give the main results of this thesis in the following.1.Optimal Switching under a Hybrid Diffusion Model and Applications to Stock TradingLet(?,F P)be a fixed probability space on which are defined a standard 1-dimensional Brownian motion B(t),t?0 and a Markov chain a(t),t ?0.Assume that B(·)and ?(·)are independent.The Markov chain is observable and takes value in a finite state space M={1,...,M}.Denote Q=(?pq)p,q?M the generator of ?(·).Let{Ft}t?0 be the completed filtration generated by B(·)and ?(·)and augmented by the null sets.Consider an 1-dimensional bybrid diffusion(?P(·),Xp,x(·))with initial state(p,x)?M×RWe consider the total payoff for a switching control Ii(·)with(?n,?n)n?1 defined by where ?>0 is the discount factor.We denote by Ai the admissible set of all switching control processes starting from the initial regime i.Our objective is to maximize the total payoff J(i,p,x,Ii(·))over Ai.Accordingly,we define the value functionand we also use vi,p(x)for v(i,p,x)interchangeably.The next theorem presents the dynamic programming principle corresponding to our optimal switching problem under a hybrid diffusion model.Theorem 0.1.Under assumptions(H1.1)-(H1.3),for any(i,p,x)?N×M×R and any stopping time 0,we haveThen,we show that the value function of our optimal switching problem is the u-nique viscosity solution to the following HJB equation(system of variational inequalities)Theorem 0.2.Assume(H1.1)-(H1.3).The value function vi,p(x)defined by(0.0.17)is a viscosity solution to the system of variational inequalities(0.0.18).Theorem 0.3.Assume(H1.1)-(H1.3).Let ui,p(x)(respectively,vi,p(x))be a viscosity subsolution(respectively,supersolution)to(0.0.18)and satisfy a linear growth condition.Then,vi,p(x)? vi,p(x).Then we assume that the Markov chain ??(·)has a two-time-scale structure,which is generated by Q?=(?(pq)?)such thatwhere Q=(?(pq))and Q=(?(pq)).Assume further that the state space of ??(·)is given by M=M1 ?…? ML,where Mk={sk1,...,skmk}for k=1,...,L and M =m1+…+ mL.In addition,Q has the following block-diagonal structure such that Qk is also a generator on Mk for k = 1,...,L.The corresponding limit system of variational inequalities isTheorem 0.4.For k = 1,...,L and l =1,...,mk,we have v(i,skl)?(x)?v(i,k)(x).More-over,v(i,k)(x)is the unique viscosity solution to the system of variational inequalities(0.0.19)for the limit optimal switching problem.Then we give two numerical examples of the stock trading problem in a regime switching market.Stock price is modeled by a switching geometric Brownian motion where ap(·)is a Markov chain taking value in M={1,2,...,M},x and p are the initial values of the stock price and Markov chain,respectively.Here,b(p),p?M,is the expected return and ?(p),p?M,represents the stock volatility.Let {?n}n?i denotes an increasing sequence of stopping times representing the switching control:A buying decision or selling decision is made at ?n.We take N={0,1}.Here the state 0 represents a flat position and 1 indicates a long position consist-ing of one share of the stock.If i=0(initially flat),then one buys at ?1 and sells at ?2,then buys at ?3 and sells at ?4,etc.On the other hand,if i=1(initially one share long),then one sells the stock at ?1,then buys at ?2 and sells at ?3,etc.Denote the trading strategy starting from regime i ?N as(?)and let Ai be the set of all these trading strategics.The objective is to choose the sequence {?n}n?1 so as to maximize where the constant K is the transaction fee associated with each buying or selling of the stock.Moreover,define v(i,p,x)=supIi(·)?Ai J(i,p,x,Ii(·)).Let us consider a new problem with the same dynamics(0.0.20)but to maximize the following objective and define v(i,p,x)= supIi(·)?Ai J(i,p,x,Ii(·)).Note that the new problem(0.0.22)has the same optimal trading rule with the original problem(0.0.21).In addition,Then we can see that the objective functional(0.0.22)will be of the following formAs a result,corresponding to the present case,the general system of variational inequal-ities(0.0.18)becomes based on which we can compute the value function and the optimal trading rule.2.Continuous-Time Mean-Variance Portfolio Selection with Random Hori-zon in an Incomplete MarketLet T>0 be the end of a finite time horizon.(?,A,{Ft}t?[0,T],P)describing the uncertainty is a complete filtered probability space.Let for m?1 and d ? 0 he an(m+d)-dimensional standard Browinian motion defined on this space.We further assume that the filtration {Ft}t?[0,T]with FT(?)A is generated by B(t),augmented by all P-null sets in A so that t ? Ft is continuous.Consider now an agent who invests at time t the amount vi(t)of the wealth x(t)in the i-th security,i = 0,1,...,m.Then,the agent's wealth x(t)with the initial endowment x0 evolves according to the following SDE:where b(t)=(?1(t)-r(t),...,?m(t)-r(t)).A portfolio v(t)is said to be admissible if u(t)?u:=LF2(0,T;Rm).We assume that the agent's exit time ? is a positive random variable measurable with respect to the sigma-algebra A,which may strictly bigger than FT generated by the underlying Brownian motion.Suppose the deadline of the investment is T after which the agent can no longer trade the assets any way.Thus,the actual exit time of the agent is T?? and his objective is to find an admissible portfolio u(t),among all such admissible portfolios whose expected terminal wealth E[x(T??)]=z,for some given z?R,so that the risk measured by the variance of the terminal wealth Var[x(T??)]=E[x(T??)-E[x(T??)]]2=E[x(T??)-z]2 is minimized.Then we have the following formulation.Definition 0.1.Under Assumptions(H2.1),(H2.2),and(H2.3),the mean-variance portfolio selection problem with random market parameters and a random horizon in an incomplete market is formulated as a linearly constrained stochastic optimal control problem,parameterized by z?R:Note that the mean-variance problem with random market parameters and a ran-dom time horizon in an incomplete market is a dynamic optimization problem with a constraint J1(u(·))= z.Here we apply the Lagrange multiplier technique to handle this constraint.For each ??R,defineThe first goal is to solve the following unconstrained problem parameterized by the Lagrange multiplier ?:This is a standard stochastic LQ optimal control problem.We introduce the following BSDEs:and(0.0.25)is the so-called stochastic Riccati equation(SRE)and(0.0.26)is the auxiliary B-SDE.We apply the theory of BMO-martingales to provide a direct and greatly simplified proof of the solvability of the two BSDEs.Theorem 0.5.Suppose Assumptions(H2.1),(H2.2),and(H2.3)hold.Then the SRE(0.0.25)admits a solution(?)satisfying k?p?K for some constants K>k>0.Moreover,?0t ?(s)'dB(s)is a BMO-martingale.Theorem 0.6.Assume(H2.1),(H2.2),and(H2.3).For a given solution(p,?)of SRE(0.0.25),BSDE(0.0.26)admits a unique solution(?)LF2(0,T;Rm+d).Moreover,we have 0<h(t)? 1 for any t ?[0,T].Furthermore,if?(t)>0,a.e.t?[0,T],then 0<h(t)<1 for all t ?[0,1).For the original mean-variance problem(0.0.23),we shall derive explicitly the effi-cient portfolios and the efficient frontier in closed forms based on the solutions of BSDEs(0.0.25)and(0.0.26).In fact,we have the followingTheorem 0.7.Let Assumptions(H2.1),(H2.2),(H2.3)and Condition(2.3.2)hold.Then Moreover,the efficient portfolio corresponding to z as a feedback control of the wealth is given by whereThe optimal wealth process is given by the solution of(2.5.1),corresponding to u?*(t).Furthermore,among all the wealth processes x(·)which satisfy the constraint E[x(T??)]=z,the optimal value of the variance of the terminal wealth Var[x(T??)is3.Stochastic Maximum Principle for Forward-Backward Regime Switching Jump Diffusion Systems and Applications to FinanceLet(?,F,{Ft}t?[0,T],P)be a complete probability space on which defined a stan-dard 1-dimensional Brownian motion,a continuous-time Markov chain,and a Poisson random measure.Markov chain ?(t)takes values in a finite space S ={?1,?2,...,?D},where D?N,?i,?RD,and the jth component of ?i is the Kronecker delta ?(ij)for each i,j=1,2,...,D.The state processes(X(t),Y(t),Z(t),Z(t,e),Z(t))?R4×RD corresponding to the control process u(t)?U(?)R are modeled by the following decoupled FBSDE:where ??R is a given constant and b,?,?,?,f are given functions with appropriate dimensions.Here we denote ?(t)=(X(t),Y(t),Z(t))for notational simplicity.Consider a performance criterion defined as:where l,g,h are given functions.The stochastic control problem is to find an optimal control u*(·)?U such that J(u*(·))= infu(·)?u J(u(·)).Let 6 denote(x,y,z)and R denote the set of all functions r:??R.Define the Hamiltonian H:(?)(?)(?).We also assume that H is differentiable with respect to ?.After in-troducing the adjoint equation(3.2.3),we get the stochastic maximum principle.Theorem 0.8.Let u*?U with a corresponding solution(X*,Y*,Z*,Z*,Z*)of(0.0.27)and suppose there exists a solution(?*,?*,?*,?*,?*)of the corresponding adjoint equa-tion(3.2.3),such that for all u?UFurthermore,we assume the following conditions hold(to simply the notations,in what follows we write(?)Condition 1.For all(?)Condition 2.For each fixed(?)exists and is a convex function of ?.Condition 3.The functions g(x,?i)and h(y)are convex for each ?i,i=1,2,…,D.Then u*is an optimal control and(X*,Y*,Z*,Z*,Z*)is the corresponding optimal state processes.Next we establish the relationship between stochastic maximum principle and dy-namic programming principle.We should reduce the cost functional(0.0.28)to J(u(·))=Y(0),corresponding to g(x,?i)= 0,h(y)= y and l(t,?,u,?i)= 0 for all i=1,2,...,D.Furthermore,we define V(t,x,?i)= infu(·)?u J(t,x,?i;u(·)).We obtain that the value function V(t,,x,?i)satisfies the following HJB equation:where the generalized Hamiltonian G associated with v?C1,2([0,T]×R)for cach ?i is defined as(3.4.2).Now we present a theorem which establishes the relationship between our stochastic maximum principle and the dynamic programming principle of Peng's type.Theorem 0.9.Assume that(?)the optimal control and(X*,Y*,Z*,Z*,Z*)be the corresponding optimal state processes.Then for all s?[t,T],we haveFurthermore,if V(t,x,?i)?C1,3([0,T]×R),we define the following processes:Then(?)are the adjoint processes and satisfy the FBSDE(3.2.3).Finally,we apply the stochastic maximum principle to solve a cash flow valuation problem with terminal wealth constraint in a Markov regime-switching jump-diffusion financial model.We can reformulate the problem as follows,where FBSDE provides a natural setup.Minimize:where c(t)is the return rate of the principal,which can be regarded as a part of the control.The wealth process X(t)of the agent is given by where u(t)is the portfolio selection of the agent,which can be regarded as the other part of the control.The utility Y(t)of the principal is given by By virtue of the properties of Markov chain and martingale representation theorem,we solve explicitly the optimal strategy of the above problem,i.e.,Theorem 0.10.The optimal strategy for the cash flow valuation with terminal wealth constraint problem(3.5.3)is given by:where ?*is given by(3.5.23),p(t,?(t))and q(t,?(t))are given by(3.5.17)and(3.5.18),respectively.4.Maximum Principle for Optimal Control of Anticipated Forward Backward Stochastic Differential Delayed Systems with Regime SwitchingLet(?,F,P)be a probability space.T>0 is a finite-time horizon.{Bt}0?t?T.is a 1-dimensional Brownian motion and {?t}0?t?T is a finite-state Markov chain with the state space given by I ={1,2,...,k}.Assume that B and ? are independent.The transition intensities are ?(i,j)for i?j with ?(i,j)nonnegative,uniformly bounded and ?(i,i)=-?j?i?(i,j).Let{Ft}0?t?T be the filtration generated by {Bt,?t}0?t?T and augmented by all P-null sets of F.Consider the following anticipated BSDE with Markov chain where the terminal conditions are at at?LF2(T,T+?;R)and bt bt?LF2(T,T+?;R).Theorem 0.11.Under the Assumptions(H4.1)and(H4.2),there exists a unique solu-tion(Yt,Zt,Vt)?LF2(0,T+?;R)×LF2(0,T+?;R)×L2(P;R)for the anticipated BSDE with Markov chain(0.0.29).We consider an optimal control problem of the following stochastic control system,which is an anticipated forward backward stochastic differential delayed equation.where Wtnt =(Wt(1)nt(1),...,Wt(k)nt(k)),and x0(t),v0(t)are deterministic functions.In the above,vt is an Ft-adapted stochastic process with values in U,and U(?)R is a nonempty,convex set.Denote by U the class of admissible control taking values in the convex domain U and satisfying E[?0T |vt|2 dt]<?.Define the cost functional as follows where l,h.,r are measurable functions.From the variational equation(4.3.1)and the estimates in Theorem 4.2,we can prove the following variational inequality.Theorem 0.12.Under the Assumptions(H4.3)-(H4.5),the following variational in-equality holdsAfter introducing the adjoint equation(4.3.5),we define the Hamilton function(?)as follows where r,r?[t,T].Then we can obtain the stochastic maximum principle.Theorem 0.13.Let ut be an optimal control and(Xt,Yt,Zt,Wt)be the corresponding trajectory.[qt,pt,kt,?t)is the unique solution of adjoint equation(4.3.5).Then,for any v?U,we haveNext we add an additional concavity assumption to obtain the sufficient condition for the optimal control.Theorem 0.14.Suppose ut?U.Let(Xt,Yt,Zt,Wt)be the corresponding trajectory and(qt,pt,kt,?t)be the solution of adjoint equation(4.3.5).If Assumptions(H4.3)-(H4.6)and(0.0.31)hold for ut,then ut is an optimal control.At last,we study an investment-consumption model under the stochastic recursive utility.Applying the maximum principle,we obtain the optimal consumption rate.Consider the following stochastic differential delayed equation with Markov chain:A consumption process ct is an Ft-adapted nonnegative process satisfying E[?0T |ct|2 dt]<?.Let U represents the set of all the consumption processes.In our investment-consumption problem,the investor wants to choose a consumption process in U to max-imize his utility.Consider the following stochastic recursive utility,which is described by a BSDE with Markovian regime-switchingwhere U(t,c,?):[0,T]×R+×??R is a given stochastic utility function.We want to find a consumption rate ?t such thatThe candidate optimal consumption rate ?t is given byWe can check the Assumptions(H4.3)-(H4.6)are satisfied,so we getTheorem 0.15.Let(qt,pt,kt,?t)be the solution of the adjoint equation(4.4.3).Suppose(4.4.4)holds.Then by Theorem 0.14 the optimal consumption rate ct is given implicitly by(0.0.32).