Stability, Approximation, and Robustness of Optimal Policies in POMDPs

In this talk, I will focus on the stability, approximation, and robustness of optimal policies in partially observable Markov decision processes (POMDPs) under discounted and average cost criteria.

I will begin with filter kernel perturbation under model change. I will establish how small perturbations in the transition or observation kernels lead to explicit, nonasymptotic bounds on both the filter kernel and the performance of the induced policies. A primary result demonstrates that an optimal policy computed under an incorrect model remains near optimal for the true model, with quantitatively bounded errors.

Building on these stability concepts, I will then talk about the average cost setting and describe conditions under which the nonlinear filter exhibits a contraction property. Under these conditions, the vanishing discounted approach yields a solution to the average cost optimality equation, guaranteeing the existence of a stationary optimal policy. Furthermore, this provides explicit bounds that quantify the influence of initial prior distribution errors and model errors on long run performance.

Finally, I will present results on implementable approximations. I will introduce refined error bounds for finite window controllers and conclude by discussing how Q learning can be utilized through either finite window or the quantization of the belief process to obtain near optimal policies for both discounted and average cost criteria.