DDG: Data-dependency gap: a new problem in the learning theory of CNNs

• Duration: 17.06.2021 - 16.06.2024
  Funding Organization: DFG
  
  Prof. Dr. Marius Kloft
  University of Kaiserslautern-Landau
  
  

  In statistical learning theory, we aim to prove theoretical guarantees for the generalizability of machine learning algorithms. The approach is usually to limit the complexity of the class of functions associated with the algorithm. If the complexity is small (compared to the number of training samples), the algorithm is guaranteed to generalize well. However, for neural networks, the complexity is often extremely large. Nonetheless, neural networks-and convolutional networks in particular-have achieved unprecedented generalization across a wide range of applications. This phenomenon cannot be explained by standard learning theory. Although a rich literature provides partial answers by analyzing the implicit regularization imposed by the training procedure, the phenomenon is by and large not well understood. In this proposal, we introduce a new viewpoint on the "surprisingly high" generalization a capability of neural networks: the data dependence gap. We argue that the essential reason for these unexplained generalization abilities may well lie in the structure of the data itself. Our central hypothesis is that the data act as regularizers in training neural networks. The goal of this proposal is to verify this hypothesis. We will conduct empirical evaluations and develop a learning theory, in the form of learning bounds depending on the structure in the data. In doing so, we will compare the weights of the trained CNNs with the taking into account the structure in the underlying data distribution. We focus on convolutional neural networks, arguably the most prominent class of practical neural networks. However, the present work may pave the way for the analysis of other classes of networks (this may happen in the second funding period of the SPP).

  Translated with www.DeepL.com/Translator (free version)