Offline Deep Q* Estimation with Diffusion Models
A novel framework is proposed that decouples operator estimation from value function learning and establishes the smoothness of $Q^*$ through the Bellman equation using the the Holder assumptions on the conditional reward and transition densities.