On the Reliability of Majority-Vote Pseudo-Labels in Test-Time Reinforcement Learning
Executive Abstract Synthesis
Test-time reinforcement learning (TTRL) adapts pretrained language models to unlabeled target-domain data using self-generated pseudo-labels, whose reliability remains poorly understood. We analyze majority-vote supervision for TTRL with online Direct Preference Optimization (DPO) and decompose its error into population-majority bias and prompt-dependent finite-rollout variation. Because population-majority bias is generally unidentifiable without external supervision, we focus on mitigating the downstream effects of finite-rollout variation. We propose \emph{Prompt-Conditional Distributionally Robust DPO (PC-Dr.DPO)}, which conditions within-prompt robust reweighting on observed vote reliability to make updates from sampling-sensitive prompts more conservative. We further introduce a cache-based mechanism to reduce self-supervised forgetting during online adaptation. Across six mathematical and knowledge-intensive reasoning benchmarks and five model backbones, PC-Dr.DPO matches or outperforms the strongest TTRL baseline in 24 of 30 settings.
Key Empirical Findings & Benchmarks
- Primary research focus: On the Reliability of Majority-Vote Pseudo-Labels in Test-Time Reinforcement Learning.
- Demonstrates empirical methodology and reproducible benchmarks across computational and analytical models.
- Autonomously indexed and synthesized by the ARGUS Scientific Intelligence Desk.
Strategic & Hardware Implementation Impact
Provides researchers, engineers, and enterprise decision-makers with early visibility into Reliability Majority Vote Pseudo.