Deep Reinforcement Learning-Based Arbitrary Beam Pattern Synthesis for Millimeter-Wave Reconfigurable Intelligent Surfaces With Discrete Phase Shifts

Abstract

Abstract Millimeter-wave reconfigurable intelligent surface (RIS) panels restricted to 1bit or 2-bit discrete phase shifts must synthesize arbitrary beam patterns from a solution space that grows exponentially with the number of reflecting elements. Existing two-dimensional finite-impulse-response filter analogies compute a solution quickly but cannot adapt to time-varying beam requirements without full recomputation, while iterative heuristics such as the artificial bee colony algorithm reduce side-lobe level (SLL) at the cost of per-frame latency exceeding the channel coherence time. A Double Deep Q-Network (DDQN) framework is proposed in which an agent learns a generalizable policy mapping target beam specifications directly to discrete RIS phase configurations. The phase assignment task is reformulated as a Markov decision process over sequential element decisions, reducing the action space from 2 N to a tractable per-element sequence, and a compound dense reward combining SLL suppression, main-lobe gain preservation, and beam-width matching is derived with a Bellman-optimality convergence guarantee. At 28 GHz with a 16 × 16 RIS, the proposed DDQN achieved an average SLL of -14.9 dB under 2-bit quantization, compared to -7.4 dB for direct superposition and -12.4 dB for the artificial bee colony algorithm, while reducing inference latency to 1.4 ms per frame, a 34× speedup over the iterative baseline. Generalization tests confirmed less than 1.1 dB SLL degradation across 200 unseen beam configurations. Ablation results identified the dueling architecture, prioritized replay, the convolutional spatial encoder, and the compound reward as the dominant contributors to the achieved gain, confirming feasibility of real-time discrete-phase beam synthesis for 6G millimeter-wave deployment.