Lightweight and Versatile Learned Optimization by Recombination of Gradient History
A lightweight and versatile learned optimizer that dynamically recombines gradient history, represented as averages over disjoint time spans, is presented, which minimizes memory cost of long history, while keeping their contributions independently accessible.