Spectral Feature Extraction for Reinforcement Learning in Financial Markets Under Nonstationary Dynamics

Abstract

Reinforcement learning (RL) offers a principled framework for adaptive trading under the nonstationary dynamics of financial markets, yet the interaction between state representation and reward design remains poorly understood. This study represents a controlled empirical comparison of six RL trading agents trained via Proximal Policy Optimization on SPY daily price data from 2015 to 2024, varying observation features (raw time-domain, spectral frequency-domain, and combined dual-stream) and reward functions (simple return-based and spectral regime-conditioned) in a factorial design. Results demonstrate that reward function formulation is the dominant determinant of test period performance; all spectral-reward conditions outperformed all simple-reward conditions on risk-adjusted metrics including Sharpe ratio, Sortino ratio, and Calmar ratio. Spectral features alone, without an adapted reward signal, produced no improvement over raw time-domain features. The best-performing condition combined dual-stream features with the spectral reward, achieving a test Sharpe of 1.36 and maximum drawdown of -12.6%, outperforming a passive buy & hold benchmark on all risk-adjusted metrics. These findings suggest that reward engineering, rather than feature engineering, is the higher-leverage design dimension for RL-based trading systems.