2026· Annual Meeting of the Association for Computational Linguistics· pp. 27665-27682· 0 citations· 37 references
Computer Science
TL;DR
This work studies value-controllable alignment through discrete condition vectors and proposes Verifiable-reward-Routed LoRA—a parameter-efficient mixture-of-experts LoRA framework enhanced with conditioned gating, which consistently out-performs prompt-based steering and multi-task PEFT baselines.
Abstract
Steerable pluralistic alignment aims to enable large language models (LLMs) to reliably adhere to diverse and potentially conflicting human values, particularly when target objectives involve multi-dimensional, compositional values. Current methods largely rely on prompt engineering or reasoning-time guidance, which often results in fragile and non-persistent con-trol once prompts are perturbed or omitted. In this work, we study value-controllable alignment through discrete condition vectors and propose Verifiable-reward-Routed LoRA—a parameter-efficient mixture-of-experts LoRA framework enhanced with conditioned gating. This gating mechanism dynamically directs the flow among multiple LoRA experts based on an input value or moral vector. To ensure that such routing leads to semantically compliant outputs, we formulate post-training as a reinforcement learning problem with verifiable rewards. We further introduce a conditional consistency reward, computed by an external model-based verifier implemented as a lightweight discriminator, and optimize the adapter parameters using GRPO. Experiments on the Touché23-valueEval (value alignment) and MIC (moral alignment) benchmarks, using two 8-billion-parameter back-bones, show that our method consistently out-performs prompt-based steering and multi-task PEFT baselines. It attains the highest over-all controllability across micro-F1, macro-F1, and Jaccard metrics—a conclusion further reinforced by human pairwise evaluations
This work proposes VEG (verbal ϵ -greedy), a novel framework that leverages external feedback as a dynamic control variable to explicitly balance exploration and exploitation within the semantic space and achieves superior accuracy compared to standard RL baselines.
Yongchang Hao, Jie Hao, Yongsheng Mei et al.· 0 citations
ReBRAC-v2 is introduced, which directly trains an exact-likelihood normalizing flow as the RL actor, combines likelihood, MSE, and MAE behavior regularization, and integrates a classification-based residual critic, staged optimization, and multi-sample test-time action selection, and ranks first in eight categories.
Reinforcement learning with verifiable rewards (RLVR) has emerged as an effective approach for improving multimodal reasoning. However, most existing methods evaluate an entire response using a binary reward based only on final-answer correctness, thereby discarding the supervision available in intermediate reasoning steps. Process reward models offer finer-grained feedback, but they typically rely on separately trained verifiers, costly chain-of-thought annotations, or online judging by large language models (LLMs). In this work, we introduce StructReward, a compute-efficient framework that provides dense reinforcement signals through structured step-level reward alignment. StructReward represents each generated solution as a sequence of reasoning steps and aligns them with process-labeled reference steps using lightweight numerical, symbolic, and lexical matching rules. The aligned labels are aggregated into a dense process reward and combined with final-answer consistency and output-validity rewards through a gated Group Relative Policy Optimization (GRPO) objective. We further recycle policy rollouts into complementary supervision for response comparison and reflective self-correction, rather than discarding them after policy updates. Separately, we use a strong LLM to rewrite sampled correct trajectories into reflection-oriented training instances, further strengthening the policy's ability to evaluate and refine its reasoning. Since reward computation is performed online without an additional learned verifier or external LLM judge, StructReward substantially reduces the computational overhead of multimodal reinforcement learning. Experimental results show that structured process supervision and rollout recycling provide an efficient path toward self-improving multimodal reasoning.
This work proposes CritiqueDriveVLM, a novel unified three-stage framework internalizing reasoning directly into the VLM, and proposes Latent Thought Distillation to overcome the latency bottleneck.
Zhaohong Liu, Hao Ye, Xianlin Zhang et al.· 1 citation
This work presents a stable and efficient training pipeline, incorporating algorithmic and system optimizations such as clipped importance sampling, training-inference ratio correction, and mixed-precision control, and proposes a structured evaluation framework across three dimensions: comprehensibility, reproducibility, and efficiency.
Xinyu Tang, Gangqiang Cao, Yurou Liu et al.· 0 citations
GFlowRL, a streamlined GFlowNet-style RL algorithm that removes the auxiliary partition network entirely while preserving the reward-distribution-matching objective, is proposed, and is the first GFlowNet-style RL algorithm to scale stably across both dense and sparse architectures.
Xiaodong Liu, Michael Xu, Jack W. Stokes et al.· 0 citations