arXiv preprint ·
Finite-Sample Sensitivity of Kernel Metrics for Generative Evaluation
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Kernel distances such as the Kernel Inception Distance (KID) and the Kernel Audio Distance (KAD) compare generated and reference samples in an embedding space, and model comparisons rest on their values. Each ships with one fixed kernel, and a reported value does not say which changes that kernel detects at the sample size used. We run each default as a permutation two-sample test on real embeddings of symbolic music, audio and images, comparing a sample with an independent sample changed by a mean shift, an isotropic contraction, or a rotation that keeps the mean and the covariance trace and is matched to the shift in transport distance. At 780 samples per side, both defaults rarely detect the rotations in five evaluation spaces, while they detect the shifts in every run. The median-bandwidth Gaussian kernel of KAD detects the contraction and regains power against rotations on image embeddings with an order of magnitude more samples. The cubic kernel of KID contains second- and third-moment information but misses the contraction in most runs, and our experiments do not establish a high-power rotation regime for it over the tested sample sizes. Narrower Gaussian bandwidths restore sensitivity in music and audio, an $\ell_1$ Laplace kernel does so for images, and aggregated tests combine both at nominal level. We recommend that a reported kernel distance state its sample budget and the changes its configuration detects at that budget.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Machine Learning
EmoNet: A Lightweight Attention-Based CNN for Speech Emotion Recognition
From Speech Tokens to Stolen Voices: Attacker-Chosen-Text Speaker Impersonation via Representation Inversion
AudioFM: Towards General Purpose Audio Representation Learning
Gangqinpu: A Multi-Genre, Multi-Task Piano Dataset with Aligned Audio, MIDI, and Musical Scores
Does Notation Matter? MusicXML Tokenisation for Symbolic Music Generation and Understanding
Get the few papers worth your time in Machine Learning every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.