Deep neural networks are key to artificial intelligence (AI). Trained neural networks can perform tasks. However, figuring out exactly how these networks work or why some are better than others is hard. Better understanding could help build smarter AI using fewer resources.
Researchers at the University of Basel have created a simple model that captures the main features of deep neural networks. Their work uses ideas from physics to make the complex behavior of neural networks easier to understand. This research is published in Physical Review Letters.
How neural networks process information
Deep neural networks have many layers of neurons. A layer is a group of neurons working together. When the network learns to identify objects in images, it processes the information step by step through these layers. Each layer should do an equal share of this work. Sometimes, though, certain layers do more, depending on how the network is built. For example, neurons can do simple linear operations, or more complex nonlinear operations. Nonlinearity makes calculations more complicated but often improves results.
Another factor is noise. Surprisingly, noise can make the network perform better. The mix of nonlinearity and noise creates complex behavior that’s hard to predict. However, when layers share the work evenly, the network works best.
To simplify this, the researchers used a model like a folding ruler. Each section of the ruler represents a layer in the network. Pulling the ruler mimics how the network processes data. Nonlinearity is like friction between the ruler’s sections, and noise is like shaking the ruler. If you pull slowly, only the first sections open, similar to a network where early layers do most of the work. Pulling fast with a little shake makes the ruler open evenly, like a network with balanced data separation. The researchers also tested models with blocks and springs, which matched real network results closely. They plan to use this approach for advanced language models and to improve AI training without relying on trial and error.