TransDubber translates English speech into Chinese that ends where the source ends and falls silent where the speaker paused. The timing is learned by the model itself: no inference-time timing controller, no padding, truncation, or time stretching.
Five test utterances, seven systems, one shared clock. Every strip starts at its clip's speech onset and uses the source's time axis, so you can see whether a system pauses where the speaker paused and stops where the speaker stopped. Click a strip to seek.
English sources from CoVoST2 (read speech) and GigaSpeech (spontaneous speech). All translations are the systems' own outputs, text and audio alike.
Timing is treated as part of the translation target. Supervision for it is built on the source's own canvas, and an autoregressive model learns it without any timing machinery at inference.
Timeline-preserving S2ST asks the target speech to follow the source on a shared absolute clock: end where it ends, go silent where it pauses. Matching total duration alone still leaves every internal pause anywhere.
Each English speech region is re-translated under a duration budget, synthesized in the speaker's voice, and placed back into its own window on the source timeline. The result has exactly the source's length, pauses and endpoint fixed.
One autoregressive model emits a pause-marked transcript, then Chinese text and speech tokens interleaved, and decides by itself when to stop. Timeline-aligned fine-tuning establishes the behavior; verifier-guided RL sharpens the timing and recovers translation quality.
Parallel speech corpora never provide this supervision, because independently recorded translations follow the rhythm of the target language. So the training targets are constructed to follow the source instead, and the model is taught to reproduce them.
Voice-activity detection finds the speaking windows and the silences between them. Gaps of at least 0.5 s become pause boundaries, and each window is transcribed; the regions are joined with a literal <pause> marker.
Every region is translated under a character budget set by its duration, with the whole utterance as context. VoxCPM2 synthesizes candidates in the speaker's voice; only those that fit the window are kept and rendered back onto the source's own room tone.
A progressive curriculum ends with supervised fine-tuning on the aligned targets and GSPO with three verifier rewards: translation quality on the text tokens, duration and pause alignment on the speech tokens.
TransDubber starts from Qwen3-ASR-1.7B, a recognition model that cannot translate or speak. Its vocabulary is extended with 6,561 speech-codec tokens, and it learns to emit the pause-marked English transcript, then Chinese text and speech interleaved five-to-ten, continuing speech until it decides the translation is over. Stage 3 is where source-timeline following appears; Stage 4 refines it and wins back part of the translation-quality cost of strict synchronization.
English-to-Chinese translation on 4,353 test utterances from CoVoST2 and GigaSpeech, compared with a strong cascaded pipeline, SeamlessM4T and SeamlessExpressive, Qwen2.5-Omni-7B, and UniSS. Full tables, human evaluation, and ablations are in the paper.
Dur@2%: share of test utterances whose translation lands within 2% of the source length. Higher is better.
Pause F1@0.4: source and system pauses (≥ 0.4 s) count as matched when their centers fall within 0.4 s, so missed and spurious pauses both hurt. Higher is better.
The same model after each training stage. Stage 2 generates natural-pace speech; Stage 3 adds the timeline-preserving targets; Stage 4 adds verifier-guided RL.
Most of the gain comes from the data. Timeline-preserving fine-tuning lifts Dur@2% from 5.2% to 48.6% and pause F1 from 0.099 to 0.518. RL then pushes them to 61.5% and 0.550, and raises COMET from 0.772 to 0.788. Starting RL from the natural-pace model instead leaves Dur@2% unchanged: RL refines timeline following, it does not create it.