--- id: lee26w_interspeech title: Diffusion Bridge Learning Between Overfitted and Underfitted Representations for speech emotion recognition authors: - Shi-wook Lee year: 2026 doi: 10.21437/Interspeech.2026-2996 isca_url: https://www.isca-archive.org/interspeech_2026/lee26w_interspeech.html pdf_url: https://www.isca-archive.org/interspeech_2026/lee26w_interspeech.pdf session: Speech Emotion Recognition and Representation 3 topics: - speech-emotion-recognition - self-supervised - multilingual category: paralinguistics-emotion labels: - multilingual - self-supervised - generative-model institutions: - National Institute of Advanced Industrial Science and Technology code: url: "" license: "" open_to_collaboration: false wiki_frontmatter: id: lee26w_interspeech category: paralinguistics-emotion labels: - multilingual - self-supervised - generative-model institutions: - National Institute of Advanced Industrial Science and Technology updated: 2026-09-29 confidence: full-paper digest: v2 source: https://doi.org/10.21437/Interspeech.2026-2996 pdf: https://www.isca-archive.org/interspeech_2026/lee26w_interspeech.pdf wiki_url: https://interspeech-2026-wiki.vercel.app/papers/lee26w_interspeech/ markdown_url: https://interspeech-2026-wiki.vercel.app/papers/lee26w_interspeech/markdown.md --- # Diffusion Bridge Learning Between Overfitted and Underfitted Representations for speech emotion recognition *Shi-wook Lee* [PDF](https://www.isca-archive.org/interspeech_2026/lee26w_interspeech.pdf) · [ISCA page](https://www.isca-archive.org/interspeech_2026/lee26w_interspeech.html) · [DOI](https://doi.org/10.21437/Interspeech.2026-2996) **Category:** `paralinguistics-emotion` · **Labels:** `multilingual`, `self-supervised`, `generative-model` **TL;DR** — Diffusion Bridge Learning is a representation-space adaptation framework that connects early-stage (underfitted) and late-stage (overfitted) encoder representations via a conditional denoising diffusion process, achieving up to +6.42 percentage points in cross-lingual speech emotion recognition weighted average recall. ## Key contributions - Identifies and leverages training-stage representation complementarity in SER, balancing early-stage smoothness/robustness with late-stage discriminability. - Proposes Diffusion Bridge Learning, a prototype-guided conditional diffusion framework learning bidirectional sample-prototype transformations. - Incorporates paired alignment, cycle-consistency constraints, and logit-pair KL regularization to stabilize stochastic geometry reshaping. - Demonstrates consistent cross-lingual generalization gains between English and Japanese corpora. ## Problem Speech emotion recognition encoders frequently overfit to dataset-specific acoustic properties, speaker traits, or recording artifacts, resulting in poor out-of-domain and cross-lingual generalization. While early-stage representations offer generic smoothness and late-stage representations provide class separation, existing models typically rely on a single checkpoint or training regime, forcing an unhelpful trade-off between robustness and discriminability. This lack of robust representation dynamics causes severe performance drops under domain shifts. ## Method The framework extracts paired representations from late-stage overfitted (domain A) and early-stage underfitted (domain B) checkpoints of a HuBERT Large backbone. To avoid unstable direct translation, it uses a prototype-mediated design: class prototypes serve as anchor points representing empirical class means within each domain. A lightweight conditional MLP acts as a shared denoiser predicting Gaussian noise, conditioned on domain and role identifiers via four discrete condition IDs. The training objective combines an epsilon-prediction loss across samples and prototypes, direct sample-prototype bridge alignment losses, cycle-consistency constraints in representation space, standard cross-entropy on a shared emotion classifier (applied to both endpoints and translations), and a logit-pair Kullback-Leibler (KL) regularization matching softened class-posterior distributions between endpoints and translations. At inference time, original and sample-to-prototype translated representations are evaluated using the shared classifier, reshaping over-specialized directions into a class-centered geometry. ## Experimental setup Experiments use three emotional speech corpora: IEMOCAP (5,531 utterances, English), MSP-IMPROV (7,798 utterances, English), and JTES (20,000 utterances, Japanese). Cross-lingual evaluation is conducted in two directions: English-to-Japanese (train on IEMOCAP + MSP-IMPROV, test on JTES) and Japanese-to-English (train on JTES, test on IEMOCAP + MSP-IMPROV). Performance is measured via Weighted Average Recall (WAR) in percentage points across five random seeds, using a HuBERT Large backbone following the SUPERB configuration. ## Results In English-to-Japanese evaluation, the high-weight setting (lambda_pair=10, lambda_cycle=1) for the overfit-to-underfit condition yields an average improvement of +6.42% WAR (p = 0.0011, max +8.10%), outperforming low-weight (+4.42%) and unconstrained settings (+3.70%). In Japanese-to-English evaluation, the same configuration achieves a +4.00% WAR gain (p = 0.0015, max +5.20%). Same-stage bridging (underfit-underfit or overfit-overfit) yields smaller, less consistent gains, indicating that bridging heterogeneous representations is essential. | Condition | Before WAR (%) | After WAR (%) | Delta (pp) | p-value | |---|---|---|---|---| | underfit – underfit (High) | 44.40 ± 1.64 | 49.34 ± 0.22 | +4.94 | 0.028 | | overfit – overfit (High) | 44.46 ± 0.15 | 44.96 ± 0.17 | +0.50 | 0.12 | | overfit (A) – underfit (High) | 43.88 ± 0.23 | 44.52 ± 0.22 | +0.64 | 0.11 | | overfit – underfit (B) (High) | 45.32 ± 0.49 | 51.74 ± 0.17 | +6.42 | 0.0011 | | overfit – underfit (B) (None) | 45.32 ± 0.49 | 49.02 ± 0.35 | +3.70 | 0.014 | ## Limitations Evaluated exclusively on cross-lingual transfer between English and Japanese, leaving multi-language or multi-domain scalability across broader linguistic families unverified. The framework relies on discrete checkpoint pairings and specific hyperparameter saturation points (lambda values around 10, 1), and overfit-side decision geometry remains comparatively rigid and resistant to stochastic smoothing. ## Why read this Speech and ML researchers focusing on out-of-distribution generalization or representation engineering will find a novel formulation of diffusion models as representation-space bridges rather than generative data pipelines. It offers a clear blueprint for combining early- and late-training dynamics to improve robustness. ## Code None released (as of this page's `updated` date). If you are an author with a repo, please claim this entry — see CONTRIBUTING.md. ## Applications Cross-lingual speech emotion recognition, empathic conversational agents, and robust paralinguistic analysis systems operating under domain shifts. ## Institutions / 機構 National Institute of Advanced Industrial Science and Technology ## Related - [Learning Emotion-discriminative Representations for Zero-Shot Cross-Lingual Speech Emotion Recognition](mi26_interspeech.md) — same problem · relatedness 2.6/3 - [Multi-Loss Learning for Speech Emotion Recognition with Energy-Adaptive Mixup and Frame-Level Attention](wang26u_interspeech.md) — same problem · relatedness 2.5/3 - [Segment-wise Embedding based Graph Attention Network for Effective Speech Emotion Recognition](song26c_interspeech.md) — same problem · relatedness 2.4/3 - [TIMBRE: Layer-Wise Cross-Lingual Speech Emotion Recognition Across 49 Layers and 26 Corpora](marchenko26_interspeech.md) — same problem · relatedness 2.4/3 - [Geometric Second-Order Feature Correlation Learning for Self-Supervised Speech Emotion Recognition](li26u_interspeech.md) — same problem · relatedness 2.3/3 All 950k paper pairs scored by TypeSafe Jev (`scripts/related/`); relatedness 0 = unrelated … 3 = directly comparable.