I’m a third-year Ph.D. student in Computer Science at the University of Texas at Austin, advised by Prof. David Harwath. My research primarily focuses on reinforcement learning, speech and audio understanding & generation, and multimodal large language models.
My research internship experience includes roles with NVIDIA’s ADLR team, where I worked with Zhifeng Kong, Sanggil Lee, Jaehyeon Kim, and Wei Ping; at Amazon, where I worked with Xiaohang Sun and Zhu Liu; and at Microsoft Research Asia, where I was mentored by Lei He and Xu Tan.
During the summer of 2023, I had the opportunity to work as a research intern at the SALT Lab at UT-Austin, collaborating with Prof. David Harwath and Prof. Eunsol Choi. Additionally, I’ve been a research Intern at the X-Lance Lab at SJTU since 2021, supervised by Prof. Xie Chen.
Download my resumé.
Ph.D. in Computer Science, 2024 - Present
The University of Texas at Austin
BSc in Electrical Engineering & Zhiyuan Honors Program of Engineering, 2020 - 2024
Shanghai Jiao Tong University
Speech-to-speech translation (S2ST) increasingly preserves what is said and how it sounds, but not necessarily when speech and silence occur. For dubbing and other temporally grounded applications, matching total duration is not enough: speech and pauses must also occur at corresponding moments. We formulate timeline-preserving S2ST, which aims to preserve the source endpoint and salient internal pauses on a shared absolute timeline. We present TransDubber, an end-to-end autoregressive system that learns this temporal structure directly from aligned target waveforms, without inference-time frame-level timing control or post-hoc temporal correction. To provide this supervision, we construct 642K exactly duration-matched English-Chinese speech pairs by translating and synthesizing target speech on source-defined temporal scaffolds. Training combines a progressive supervised curriculum with verifier-guided reinforcement learning to balance translation quality and temporal alignment. On a 4,353-utterance test set spanning read and spontaneous speech, TransDubber increases the proportion of translations within 2% of the source duration from 25.6% to 61.5% and pause-alignment F1 from 0.118 to 0.550 compared with UniSS, while maintaining competitive translation quality. Our analysis shows that aligned supervision establishes timeline following, while reinforcement learning refines temporal precision and partially offsets the translation-quality cost of strict synchronization. Together, these results demonstrate the effectiveness of our data and training framework for timeline-preserving S2ST.
Spatial sound reasoning is a fundamental human skill, enabling us to navigate and interpret our surroundings based on sound. In this paper we present BAT, which combines the spatial sound perception ability of a binaural acoustic scene analysis model with the natural language reasoning capabilities of a large language model (LLM) to replicate this innate ability. To address the lack of existing datasets of in-the-wild spatial sounds, we synthesized a binaural audio dataset using AudioSet and SoundSpaces 2.0. Next, we developed SpatialSoundQA, a spatial sound-based question-answering dataset, offering a range of QA tasks that train BAT in various aspects of spatial sound perception and reasoning. The acoustic front end encoder of BAT is a novel spatial audio encoder named Spatial Audio Spectrogram Transformer, or Spatial-AST, which by itself achieves strong performance across sound event detection, spatial localization, and distance estimation. By integrating Spatial-AST with LLaMA-2 7B model, BAT transcends standard Sound Event Localization and Detection (SELD) tasks, enabling the model to reason about the relationships between the sounds in its environment. Our experiments demonstrate BAT’s superior performance on both spatial sound perception and reasoning, showcasing the immense potential of LLMs in navigating and interpreting complex spatial audio environments.