Shared-Private Cross-Modal Discrepancy Learning for Imbalanced Multi-Source Audio-Visual Deepfake Detection

Abstract

Audio-visual deepfake detectors are often trained under a single dataset or manipulation regime, but multi-source training collections may contain heterogeneous attacks and uneven source distributions. Training separate detectors for each source is inefficient, while a single shared detector may be biased toward dominant source-specific patterns. We study known-source imbalanced multi-source audio-visual deepfake detection using FakeAVCeleb and LAV-DF, where source identity is supplied during training and inference. To address this setting, we build on the Multi-Scale Cross-Modal Transformer Encoder (MSCT) and propose Pre-MSCT Shared-Private Feature Fusion. Our method routes \texttt{[CLS]}-augmented audio and visual tokens into a shared stream and source-specific private streams before the first MSCT block, while retaining its internal multi-scale self-attention (MSSA), differential cross-modal attention (DCA), and feed-forward operations. The shared stream models cross-source reusable evidence, while the private streams retain source-dependent residual variations. A domain-aware gated residual fusion module injects selected private evidence into the shared representation for the main fused prediction. Experiments under two controlled source-composition protocols show improved macro and worst-source AUC compared with the MRDF and MSCT baselines. These results support shared-private cross-modal discrepancy learning in the evaluated two-source setting with known source identity.

The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.

More in Sound

• arXiv preprint · 18 October 2026

  DF-PALM: Dynamic Fusion Prompt Learning for Audio--Language Models

  Yulin Sun, Shenghua Li, Yong Dou and 3 others
• arXiv preprint · 19 October 2026

  Manifold-Constrained Residual Adaptation with Frame-Adaptive Layer Aggregation for Speaker Verification

  Zezhong Jin, Xiaoyu Wang, Zhe Li and 4 others
• arXiv preprint · 19 October 2026

  Multilingual Voicemail and Answering Machine Detection via Pretrained Audio Embeddings and Gradient Boosting Machines

  Kemal Altwlkany and Muamer Paric
• arXiv preprint · 18 October 2026

  CASE: Complementary Articulation-Aware Speech Encoder to Enhance Speech-Driven 3D Facial Animation

  Jaehong Myung, Seungho Eum, Taegon Kim and 2 others
• arXiv preprint · 19 October 2026

  AudioScorer: Learning Question-Aware Token Relevance for Large Audio Language Models

  Marwan Farag and Bin Yang

All new papers in Sound