Fine-tunning of RL models via local policy iteration

Inspired by recent challenges in fine-tuning reinforcement learning (RL) models, we introduce a fine-tuning method employing a local adaptation of Policy Iteration (PI). Our proposed algorithm can be deployed over tasks with large (or even infinite) state spaces, while preserving the existing knowledge of the model. We propose two local variants of the PI algorithm: One utilizing a fixed lookahead (or even no lookahead), and another utilizing an adaptive lookahead. We provide explicit iteration complexity bounds for both local PI algorithms. We empirically validate the efficacy of our local algorithms, and discuss several implications and challenges of applying local fine-tuning methods to deep RL models.