2023-03-24: Riu Rodríguez Sakamoto

Sat, 21 Oct 2023 21:24:08 +0100

Riu Rodríguez Sakamoto

From discounted value to discounted state

Abstract

The discounted value is a concept in Reinforcement Learning methods such as Value Iteration that allows to aggregate expected future rewards. It can be formalized as lens precomposition with a value function (which is a counit). We give some progress towards another class of RL methods that are based on parametrized policies, where this composition happens without the need of a counit, leading to a "discounted state" of a dynamical system. This has some resemblance to a List algebra, taken over a bicategory of linear relations of a vector space instead of over Set. The discounted state is used in the literature to prove the Policy Gradient Theorem, which we interpret as an Euler integration for noncommutative operators.

2023-03-24 15:00:00 UTC
LT711 and Online