Text-to-music · Pairwise reward modeling · Reinforcement learning

DuetJudge

Reinforcement Learning for Music Generation with Pairwise Reward Modeling

Zhisheng Zheng1* Zhifeng Kong2 Sang-gil Lee2 Jaehyeon Kim2 Arushi Goel2 David Harwath1 Mohammad Shoeybi2 Bryan Catanzaro2 Wei Ping2†

1The University of Texas at Austin 2NVIDIA

*Work done during an internship at NVIDIA †Leads the effort

Correspondence: zszheng@utexas.edu, zkong@nvidia.com, sanggill@nvidia.com, wping@nvidia.com

arXiv soon Listen to samples BibTeX

Listen

Hear the difference

Hear it before you read it. Generations for Song Describer captions from the evaluation set: Base is the generator before RL, the other rows are the same generator after RL with different training rewards.

TL;DR

Compare, don’t score.

DuetJudge listens to two clips conditioned on the same caption and answers A, B, or Tie — caption alignment first, perceptual quality second. Aggregating its pairwise probabilities gives a reinforcement-learning reward that improves Audex-2B, Audex-30B-A3B, and diffusion-based ETTA on all five automatic metrics and both human ratings.

01 · Standalone judge

71% agreement with human preferences

Exact-label agreement (A, B, or Tie) on same-caption Audex-2B pairs, in %. Pointwise scores are turned into decisions with a per-baseline tie threshold.

Audiobox-CE42.0
Audiobox-PQ46.0
CMI-RM Musicality42.0
CMI-RM Alignment41.0
MuQ-MuLan54.0
TuneJury53.0
DuetJudge71.0

Also best on the tie subset (60.0% vs. 50.0% for the best baseline) and the non-tie subset (73.8% vs. 58.8%).

02 · Alignment first

Right priority under conflict

Accuracy when the better-aligned clip is deliberately degraded, in %. Audio-only aesthetics pick the cleaner clip; DuetJudge keeps alignment first.

Audiobox-PQ12.5
Audiobox-CE18.8
CMI-RM Musicality26.3
CMI-RM Alignment28.7
MuQ-MuLan46.3
TuneJury52.5
DuetJudge66.3

Not by ignoring quality: on same-source degradations DuetJudge is 86.3% accurate (Audiobox-PQ 95.0%, MuQ-MuLan 42.5%).

03 · As an RL reward

Listeners rate it higher

Human overall quality (OVL, 1–5) of Audex-2B after RL with each training reward; pointwise scores are discretized into metric-specific bins (Table 2).

Base (no RL)2.697
Audiobox-CE2.412
Audiobox-PQ2.912
CMI-RM Musicality2.636
CMI-RM Alignment2.606
MuQ-MuLan2.697
TuneJury2.971
DuetJudge3.147

Caption relevance (REL) rises too, 2.606 → 2.927. DuetJudge is the only reward that improves all seven scores over Base.

04 · Same judge, same optimizer

Keep the comparison pairwise.

Converting the pairwise judgments into a scalar Bradley–Terry reward model (DuetJudge-BTRM) and optimizing Audex-2B with the same GSPO setup gives lower scores on all five automatic metrics than using DuetJudge’s pairwise probabilities directly.

1.303vs. 0.998 with the scalar rewardCMI-RM Musicality
1.435vs. 1.229 with the scalar rewardCMI-RM Alignment
0.296vs. 0.269 with the scalar rewardMuQ-MuLan

Table 3 · Audex-2B · GSPO · DuetJudge-BTRM converts the same pairwise judgments into a scalar score per caption–audio pair.

Why

Judge the pair, not the clip.

A polished, upbeat pop arrangement is not a successful response to “a slow, mournful solo cello piece,” yet matching the requested instrument and mood does not make audible artifacts irrelevant. A useful judge must consider both, with an explicit priority: caption alignment first, perceptual quality second.

Diagram: pointwise evaluators map one clip to one number; DuetJudge takes the caption plus two clips and outputs A, B, or Tie.
Pointwise scoring vs. direct pairwise judgment. Top: pointwise reward models evaluate each clip independently. Bottom: DuetJudge compares two candidates conditioned on the same caption, alignment first and quality second, with an explicit Tie option. Candidate B wins because it fulfills the requested instrument, tempo, and mood despite its rougher production.
Problem

Pointwise scores never see the alternative

Existing evaluators score one clip at a time, even the ones trained on pairwise labels. A direct judge can inspect both clips and decide which audible differences matter under the caption.

Criterion

Make the priority explicit

Overall listening appeal need not coincide with request fulfillment, so DuetJudge learns a task-specific rule: a meaningful advantage in caption alignment decides, and perceptual quality separates comparably aligned candidates.

Payoff

A reward you can optimize against

The judge stays frozen while a generator is trained; comparisons among the generator’s own same-caption samples become the reward. Audex-2B, Audex-30B-A3B, and diffusion-based ETTA all improve.

Paper abstractFull text

Improving text-to-music models through reinforcement learning (RL) requires a reward model that can clearly quantify the quality of a text–music pair. In this domain, this usually includes (i) how high-quality the outputs sound, and (ii) how well they align with the provided text. We introduce DuetJudge, an on-policy pairwise reward model that learns to directly compare two instrumental pieces conditioned on the same caption, without assigning independent scalar scores to each. DuetJudge is functionally a ternary classifier that outputs either one piece wins or they are a tie. Particularly, DuetJudge prioritizes text–music alignment, and then considers perceptual quality of the music. DuetJudge achieves 71% agreement with human preferences on our internal test set, outperforming all existing score-based baselines. To use DuetJudge as a reward model in text-to-music RL, we (i) design a reward score based on aggregated pairwise preference probabilities, and (ii) introduce classifier-free guidance (CFG)-aware policy optimization for text-to-music models trained with CFG. We test DuetJudge using multiple in-distribution and out-of-distribution text-to-music models, and find systematic improvements in both automatic metrics and human ratings. Ablation studies further favor direct pairwise rewards over a Bradley–Terry scalar baseline under the same training settings. These results confirm the effectiveness of our proposed DuetJudge, our data curation and augmentation methods, our reward model training recipes, and our text-to-music RL training recipes.

01 · The judge

Compare both clips, answer A, B or Tie

Built on Audex-2B with decision-only supervision: one context holds the caption and both clips, and only the A / B / Tie decision carries loss. No scalar head, no per-clip score.

Input “A slow, mournful solo cello piece”
clip Aclip B
DuetJudge single inference run · predicts A / B / Tie
Learned preference criterion
  1. 1
    Caption alignmenthas priority
  2. 2
    Perceptual qualityif alignment is comparable
  3. 3
    Tieif both are comparable, or evidence is insufficient
p(A), p(B), p(Tie)
A
B
Tie
B is better
One criterion, three steps
The same rule drives the judge prompt, the expert labels, and the human-evaluation protocol.
Meaningful caption- alignment advantage? yes That candidate wins regardless of polish no — alignment comparable Perceptual-quality difference? yes Better-sounding wins quality decides no — comparable in both TIE evidence insufficient → Tie
1.1

Three supervision families

Holistic pairs teach the whole rule; the other two isolate one factor. Counts are before order augmentation.

Holistic preference

“Energetic funk groove with brass stabs and slap bass”

A · realrecording
B · syntheticgenerated
ABTie← experts, alignment-first rule

Teaches: overall preference on real–real, real–synthetic, and synthetic–synthetic pairs.

163,327 pairs · 908 h

Fine-grained alignment

one attribute only: “Tempo: 120 BPM”

A · matches ✓120 BPM
B · mismatch ✗82 BPM
ABTie← label by construction

Teaches: did the clip follow the prompt? Tempo, key, chord, instrument, genre, mood.

65,152 pairs · 362 h

Perceptual quality

same caption, same source clip for A and B

A · originalclean
B · degradedcodec · low-pass · noise · clipping
ABTie← label by construction

Teaches: which clip sounds better when alignment is held fixed.

59,239 pairs · 329 h

1.2

Two-stage training

  1. StartAudex-2B

    Audio-language model that both understands and generates music.

  2. Stage IMusic understanding

    Supervised fine-tuning on music, audio, and text instruction data.

  3. Stage IIPairwise specialization

    Decision-only training on the three families, weighted by behavior rather than size.

  4. ResultFrozen DuetJudge

    Used unchanged as the reward for every RL run below.

02 · From judgments to rewards

Rewards from pairs

The judge never changes. For every caption the generator samples a group of clips, DuetJudge compares every pair in both orders, and each clip’s reward is how often it is preferred over its siblings. No reference recordings are needed.

From pairwise judgments to a reward
Order-averaged soft margins against every same-caption opponent
caption c Generator πθAudex (AR) · ETTA (diffusion) G candidates x1 x2 xG ⋮ DuetJudge · frozenevery pair (i, j), both orders p(A), p(B), p(Tie)soft margin mij reward rimeanj mij policy update · GSPO for Audex, DiffusionNFT for ETTA
Soft, not hard. A win counts +1, a loss −1, a tie 0, weighted by the judge’s probabilities, so an unsure judge produces a small margin instead of a coin flip.
Both orders. Each pair is judged as (A, B) and (B, A); averaging removes position bias.
Relative, not absolute. Opponents are the policy’s own same-caption samples, so rewards sum to zero within a group and need no reference audio.
Margin matrix four candidates as an example
x1x2x3x4ri
x1–m12m13m14r1
x2−m12–m23m24r2
x3−m13−m23–m34r3
x4−m14−m24−m34–r4

antisymmetric · each reward is a row mean · rewards sum to 0

CFG-aware policy optimization for Audex
Optimize the policy you sample from, and hold its baseline still
caption c null prompt ∅ Audex θ zc conditional z∅ null CFG mixzcfg = γ zc + (1 − γ) z∅ guided policyπcfg rollouts DuetJudgefrozen update θ reward gradient cut null conditional guided γ × gap δ (1−γ)δ shared θ:the baseline drifts,guidance amplifies it frozen SFT reference anchor null anchor guided

Audex samples with classifier-free guidance, so the guided policy is what GSPO updates and scores. The reward gradient returns through the conditional branch only, but both branches share one model: the null baseline still drifts and guidance amplifies the drift. Anchor both the guided policy and the null baseline to the frozen SFT reference.

Transfer to a diffusion generator: ETTA
Same judge, same reward; the generator learns from finished samples
caption c ETTAdiffusion with CFG G candidates decode DuetJudge · frozennever saw ETTA audio reward ri weight w 10.50 no update clean latent z0 re-noise, fresh noise zt guided prediction ẑ0 pull toward z0 weight w push away weight 1−w no sampler trajectories needed above average: pull average: no update below average: push gradient ∝ (2w−1) anchor: frozen initial ETTA

ETTA’s audio never appears in the judge’s training data. Rewards become preference weights between 0 and 1; DiffusionNFT re-noises each finished sample, pulls the guided prediction toward it with weight w and pushes away with 1 − w, anchored to the frozen initial ETTA. Same frozen judge, same reward; only the update rule changes.

03 · Results

Judge, then reward

3.1

As a judge: 71.0% agreement with human preferences

Agreement with human A / B / Tie labels
Same-caption Audex-2B pairs, percent. The best pointwise evaluator reaches 54.0.

DuetJudge also leads on the tie subset (60.0 vs. 50.0) and the non-tie subset (73.8 vs. 58.8). Same-caption, same-generator pairs are exactly the regime of the RL rollouts.

Alignment first, without going deaf to quality
Two controlled tests on the same pairs, percent correct
Conflict testDegrade the better-aligned winner. Still picked?
aligned + degradedvs.less aligned, clean
Quality testDegraded copy vs. original. Original picked?
degraded copyvs.original

Only DuetJudge is upper-right: it wins the conflicts (66.3) while still hearing the defects (86.3). Audio-only aesthetics hear defects but are alignment-blind; text-similarity scores are near chance on quality.

3.2

As a reward: every metric up, on three generators

Each pointwise evaluator is also used as a training reward for Audex-2B and ETTA. DuetJudge is the only reward that improves all five automatic metrics and both human ratings over Base on Audex-2B, and the same frozen judge improves Audex-30B-A3B and ETTA on all seven scores.

Training reward Audiobox Aesthetics ↑ (1–10) CMI-RM ↑ (1–5) MuQ ↑ Human Evaluation ↑ (1–5)
CE PQ Musicality Alignment OVL REL
Reference Audio 7.368 7.862 1.883 1.698 0.301 3.765 3.606
Base (no RL) 6.852 7.257 0.651 1.006 0.209 2.697 2.606
Audiobox-CE 8.137† 8.022 0.718 1.025 0.142↓ 2.412↓ 2.206↓
Audiobox-PQ 7.944 8.478† 0.831 1.041 0.151↓ 2.912 2.697
CMI-RM Musicality 8.009 8.342 1.961† 1.643 0.168↓ 2.636↓ 2.312↓
CMI-RM Alignment 7.697 7.782 1.900 2.067† 0.076↓ 2.606↓ 2.656
MuQ-MuLan 7.367 7.737 0.793 1.082 0.264† 2.697 3.030
TuneJury 7.994 8.205 1.283 1.200 0.145↓ 2.971 2.758
DuetJudge (ours) 7.717 8.123 1.303 1.435 0.296 3.147 2.927

↓ marks scores below Base; † marks the training-reward metric; bold is best among rewards. The only row that improves all seven: OVL 2.697 → 3.147, REL 2.606 → 2.927.

3.3

Keep the comparison pairwise

Under the same GSPO setup on Audex-2B, direct pairwise rewards beat a Bradley–Terry scalar reward model built from the same judgments (DuetJudge-BTRM) on all five automatic metrics. DPO on the judgment pairs helps too, but moving online with a scalar reward loses alignment.

Pairwise vs. scalar under the same GSPO
Audex-2B after RL; higher is better
CMI-RM Musicality
CMI-RM Alignment

Pairwise beats BTRM on all five metrics. Online alone is not enough: going from offline DPO to online GSPO with the scalar reward loses alignment.

Pointwise vs. pairwise rewards under GSPO, with DPO as an additional comparison, for Audex-2B. DuetJudge-BTRM assigns a scalar score to each caption–audio pair using Bradley–Terry reward modeling; DPO uses the non-tie judgment pairs.
Optimization Reward model Audiobox Aesthetics ↑ (1–10) CMI-RM ↑ (1–5) MuQ ↑
CE PQ Musicality Alignment
Base (no RL)–6.8527.2570.6511.0060.209
DPO–7.2927.7010.9871.3250.288
GSPODuetJudge-BTRM7.6988.0350.9981.2290.269
GSPODuetJudge7.7178.1231.3031.4350.296
Which parts of the recipe matter?Ablations

Every ablation lowers Audiobox-PQ, CMI-RM Musicality, CMI-RM Alignment, and MuQ: explicit tie labels, swap augmentation, order averaging, soft probability margins, and comparing against the policy’s own same-caption samples all contribute.

Ablations of judge training and reward construction for Audex-2B RL. The full recipe uses tie-supervised, swap-augmented judge training and intra-group comparisons with order-averaged probability-margin rewards.
Configuration Audiobox Aesthetics ↑ (1–10) CMI-RM ↑ (1–5) MuQ ↑
CE PQ Musicality Alignment
Full recipe (default)7.7178.1231.3031.4350.296
Judge-training changes
w/o tie-labeled pairs7.7718.1191.1331.3150.278
w/o swap augmentation7.6708.0191.0541.2520.289
Reward-construction changes
w/o order averaging7.7478.0201.2481.4060.295
Hard-decision rewards7.5887.9391.1651.3680.283
Joint changes
w/o swap augmentation and order averaging7.6828.0060.9441.1770.264
w/o swap augmentation; hard-decision rewards7.8188.1071.0591.2350.262
Comparison-set changes
Real-audio references only7.5047.7531.1071.3450.293
Intra-group + real-audio references7.7218.0511.1731.3800.283

Citation

Cite

@misc{zheng2026duetjudge,
  title         = {Reinforcement Learning for Music Generation with Pairwise Reward Modeling},
  author        = {Zheng, Zhisheng and Kong, Zhifeng and Lee, Sang-gil and Kim, Jaehyeon and
                   Goel, Arushi and Harwath, David and Shoeybi, Mohammad and Catanzaro, Bryan and Ping, Wei},
  year          = {2026},
  eprint        = {XXXX.XXXXX},
  archivePrefix = {arXiv},
  note          = {arXiv identifier to be added}
}