[KDD 2020] Measuring Model Complexity of Neural Networks with Curve Activation Functions
Aug 13, 20207 views
It is fundamental to measure model complexity of deep neural,networks. The existing literature on model complexity mainly focuses on neural networks with piecewise linear activation functions.,Model complexity of neural networks with general curve activation,functions remains an open problem. To tackle the challenge, in,this paper, we first propose,linear approximation neural network,(,LANN,for short), a piecewise linear framework to approximate a,given deep model with curve activation function. LANN constructs,individual piecewise linear approximation for the activation function of each neuron, and minimizes the number of linear regions,to satisfy a required approximation degree. Then, we analyze the,upper bound of the number of linear regions formed by LANNs, and,derive the complexity measure based on the upper bound. To examine the usefulness of the complexity measure, we experimentally,explore the training process of neural networks and detect overfitting. Our results demonstrate that the occurrence of overfitting is,positively correlated with the increase of model complexity during,training. We find that the,𝐿,1,and,𝐿,2,regularizations suppress the,increase of model complexity. Finally, we propose two approaches,to prevent overfitting by directly constraining model complexity,,namely neuron pruning and customized,𝐿,1,regularization.