Under stability and regularity conditions on value function gradients, Bellman residual derivatives, and fitted critics, it is proved an $\mathcal{O}(\log K)$ regret bound with constants uniform in $\delta$.
Abstract
High-frequency control problems are often studied through time discretization, but refining the timestep $\delta$ increases the discrete horizon $H=T/\delta$ and can make standard finite-horizon regret guarantees deteriorate even when the physical horizon $T$ is fixed. We study episodic value-based reinforcement learning for Euler discretizations of controlled diffusions. The performance is measured as the cumulative regret over $K$ episodes relative to the optimal policy of the corresponding Euler model. Under stability and regularity conditions on value function gradients, Bellman residual derivatives, and fitted critics, we prove an $\mathcal{O}(\log K)$ regret bound with constants uniform in $\delta$. For dynamics with drift linear in unknown parameters and unknown constant diffusion covariance, we verify localized versions of these conditions for a constrained, model-based controlled-diffusion fitted-$Q$ algorithm (CD-FQ) and obtain high-probability $\mathcal{O}(\log K\log\log K)$ regret for fixed and finite $T$, again with constants uniform in $\delta$.
We study the last iterate of stochastic gradient descent (SGD) on streaming least squares with discretely distributed features (Slock's model), which decouples the dynamics across eigendirections. Under power-law source and capacity conditions, we derive the exact finite-time risk and asymptotic equivalents of constant...
Noé Colomban, F. Bach, Fabian Schaipp· 0 citations
We study episodic continuous-time control with an unknown vector control gain, scalar state, quadratic action cost, and smooth convex terminal cost. In the scalar Gaussian experiment, let $\Delta(H)$ denote the minimax improvement over zero control and set $\delta=\sqrt2H^2-1$. We prove the critical law $$ \Delta(H)\as...
The exact exponent of linear regret deviations on two-armed Gaussian instances with common variance is determined and a matching lower bound among policies satisfying the same expected-regret budget on every such Gaussian instance is established.
A scalable model-free actor--critic algorithm is proposed, establishing finite-sample critic and near-stationarity guarantees for a linear LSTD critic and a more expressive neural TD parameterization is introduced.
Salah Chikhi, A. Chaoui, A. Ozdaglar et al.· 0 citations
It is proved that, under aperiodicity and in the near-on-policy regime, the operator underlying LKQL improves the policy evaluation convergence rate over its truncated counterpart for sufficiently large $n$, and that LKQL itself converges almost surely to the optimal values in finite Markov decision processes (MDPs) un...
Tolga Ok, A. S. Kolarijani, Peyman Mohajerin Esfahani et al.· arXiv.org· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.