Converting POMDPs into MDPs using History Representation

Abstract

Defining utility functions like the V and Q functions is essential for developing reinforcement learning (RL) solutions for POMDPs. Ideally, we want to define these functions over histories of observations and actions, which the agent can observe. However, the number of possible histories grows exponentially with the trajectory length, making it impractical to reliably estimate history-based utility functions. A common solution is to construct a compact representation of the history. Recently, with the resurgence of deep learning, researchers use recurrent neural networks to learn low-dimensional history representations. Conversion from POMDPs to MDPs using history representation appears to be so effective and seamless that it is common to see a theory-practice gap in deep RL papers, where an algorithm is theoretically formulated in an MDP setting but is empirically evaluated on POMDP tasks without any justifications on why it would work in the latter setting. This document provides a justification for the conversion from POMDPs to MDPs using history representation.