DCT Lexicon · Governing Dynamics
Occupation Measures
An occupation measure records how often a policy visits each state–action pair. In these variables, stochastic control becomes a linear program, and nonconvex policy search becomes convex.
DCT Governing Dynamics
In depth
Formal statement
For a discounted problem with initial law , the discounted occupation measure of a policy satisfies a linear balance equation, and every such measure comes from a policy.
DCT Governing Dynamics: the element's drawer (/governing-dynamics)
Linear programming duality for control
The occupation-measure program and the Bellman inequality program s.t. are dual linear programs with equal values; an optimal measure disintegrates into an optimal (possibly randomised) stationary policy.
DCT Governing Dynamics: the element's drawer (/governing-dynamics)
In DCT
Constraints on expected costs (an average draw on the line, a budget on hiring) are linear in the occupation measure, so constrained stochastic control stays convex. This is the route to certifying constrained policies with moment and semidefinite relaxations.
DCT Governing Dynamics: the element's drawer (/governing-dynamics)
Limits
Constrained optima may require randomised policies. In continuous spaces the program is infinite-dimensional; moment–SOS hierarchies converge but grow quickly in size.
DCT Governing Dynamics: the element's drawer (/governing-dynamics)
Go next
- Dynamic Risk Measures: Risk constraints on the occupation measure lead to dynamic risk measures (Pathway 9).
- Fenchel–Rockafellar Duality and Dual Bounds: Its dual is the Bellman inequality program.
- Chance Constraints and Certificates: Chance constraints are measure constraints on the visited set.
DCT Governing Dynamics: the element's drawer (/governing-dynamics)
References
- Manne (1960)
- Altman (1999), Constrained Markov Decision Processes
- Lasserre (2010), Moments, Positive Polynomials and Their Applications
DCT Governing Dynamics: the element's drawer (/governing-dynamics)
In the book: Vol. II, Ch. 7: Dynamic Programming for Enterprise Systems · Vol. II, Ch. 5: Dynamic Enterprise Optimization
Related: Dynamic Risk Measures · Fenchel–Rockafellar Duality and Dual Bounds · Chance Constraints and Certificates
GD.15


