arXiv preprint ·
A Class of Rényi-based Concentration Metrics for Kolmogorov-Arnold Geometry in Multilayer Perceptrons
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
The interpretability of artificial intelligence models has become a central concern, especially for black-box models such as multilayer perceptrons (MLPs). Kolmogorov-Arnold Geometry has recently emerged as a framework for analysing the internal representations of these networks, by examining how the hidden layer locally responds to changes in the input through the minors of its Jacobian. The degree to which a few of these minors dominate the rest, known as their concentration, has so far been measured with the Participation Ratio. Although this metric has useful properties, its maximum value depends on the size of the network, so a given value cannot be read as high or low concentration without reference to the architecture. To overcome this limitation, in this paper we propose the $α$-RC metrics, a family of Rényi-based concentration measures that compare the normalized magnitudes of the minors with the uniform distribution through the Rényi divergence of order $α$. All the metrics in the family are bounded between 0 and 1 regardless of the architecture, with 0 indicating that the magnitude is evenly spread over the minors and 1 that it is carried by a single one. Moreover, the metrics are non-decreasing in $α$, and for $α$ = 2 they reduce to an explicit transformation of the Participation Ratio. Computational experiments with single-hidden-layer MLPs trained on a synthetic dataset show the benefits of our proposal.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Information Theory
Private Evaluation of High-Dimensional Functions
Information Estimation with Discrete Diffusion
Information Theory of Random Composition Trees
Efficient Agnostic Learning of Univariate Densities Under Local Differential Privacy
Query-Dependent Reading for Quantized Inner-Product Estimation: A Minimax Separation
Get the few papers worth your time in Information Theory every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.