source: arxiv statistics ml: function-counting theory for low-dimensional data structures
level: research
deep learning often works well because real-world data, though high-dimensional in representation, has low-dimensional structure. this paper builds on cover's 1965 function-counting theory to study binary classification on such data. cover's original theory assumes points are in general position, ignoring any underlying structure. the new work refines this assumption to account for low-dimensionality, deriving new counts of possible labelings (dichotomies) that reflect the data's structure.
the framework extends cover's concepts of separation capacity and generalization to the low-dimensional setting. separation capacity is the number of points a model can correctly classify in all possible ways. generalization refers to how well a model performs on unseen data. by incorporating data structure, the analysis shows how these properties change. for example, when data lies on a low-dimensional manifold, the effective capacity may be lower than in the ambient space, affecting model design and training.
the results provide a theoretical lens to understand why deep networks succeed on structured data like images or text. the dichotomy counts depend on the intrinsic dimension rather than the embedding dimension. this can guide choices in model complexity and sample sizes. the work is mathematical, with proofs and derivations, but the implications are practical: knowing the data's structure helps predict when a model will generalize well or overfit.
why it matters: it gives a theoretical basis for how data structure influences learning, helping practitioners choose models and estimate sample needs for low-dimensional data.
source: arxiv statistics ml: function-counting theory for low-dimensional data structures