The neural coding framework for learning generative models
Introduction
For decades, backpropagation (backprop) has been the dominant engine powering deep learning. It enables neural networks to learn by propagating errors backward through layers, adjusting weights to minimize prediction mistakes. Yet this algorithm faces well-known criticisms from a biological perspective: it requires non-local error signals, symmetric forward and backward weights (the "weight transport problem"), and locks updates until the entire backward pass completes.
In a 2022 paper published in Nature Communications, researchers Alexander Ororbia and Daniel Kifer propose a compelling alternative: the Neural Generative Coding (NGC) framework. Inspired by predictive processing, a leading theory in neuroscience that views the brain as a hierarchical prediction machine constantly minimizing errors between expected and actual sensory input, NGC offers a way to train generative neural networks using only local, biologically plausible updates.
What is Neural Generative Coding?
At its core, NGC reframes learning as a "generate-then-correct" process. Instead of global error backpropagation, the framework uses two types of neurons:
State neurons: These represent the model's internal beliefs or latent representations at each hierarchical layer.
Error neurons: These compute local mismatches (prediction errors) between what a layer expects and what it actually receives from the layer below.
The network operates through settling dynamics, an iterative process where states are refined over multiple time steps via top-down predictions and bottom-up corrections. Learning then occurs through simple, Hebbian-like local rules: weights update based on the correlation between presynaptic activity and postsynaptic error signals. No derivatives of activation functions are needed, and no global feedback pathway is required.
The authors implement this idea as Generative Neural Coding Networks (GNCNs), with several variants:
Basic versions that recover classic predictive coding models (e.g., Rao-Ballard or Friston-style).
Advanced versions with learnable error synapses and lateral connections that promote sparsity through local competition between neurons (self-excitation + inhibition within groups).
Mathematically, predictions flow downward, errors flow upward locally, and state updates incorporate leak terms, error pressures, and lateral interactions. This setup naturally supports generative modeling, learning to approximate the probability distribution of data so the network can reconstruct inputs, generate new samples, or complete patterns.
Key Experimental Results
The team tested GNCNs on standard benchmarks: MNIST, Fashion-MNIST, KMNIST, and Caltech 101 Silhouettes (binary image datasets).
Highlights include:
Reconstruction and density estimation: NGC models achieved competitive or better negative log-likelihood and reconstruction error compared to Variational Autoencoders (VAEs) and other backprop-based generative models.
Data efficiency: NGC often converged faster and generalized better with fewer training examples or earlier in training.
Pattern completion (inpainting): When half the image was masked, NGC significantly outperformed baselines in reconstructing the missing parts.
Downstream classification: Even though trained unsupervised, the learned representations enabled better or comparable classification accuracy than models explicitly trained for the task.
Sparsity and representation quality: Lateral inhibition produced cleaner, more structured latent spaces with clearer class separation (visible in t-SNE plots).
Sample generation: Generated images were realistic and competitive with stronger baselines like GAN-autoencoder hybrids.
Importantly, while NGC requires more computation per sample due to iterative settling, it often needs fewer overall epochs, suggesting potential efficiency gains in the long run.
The code for all experiments is openly available via the ngc-learn Python library, making the work highly reproducible.
Strengths and Limitations
Strengths:
Strong biological plausibility: local learning rules, derivative-free updates, support for recurrent/lateral dynamics.
Unifies several predictive coding theories under one practical framework.
Excels at tasks requiring robust internal models (e.g., completion, few-shot generalization).
Opens the door to more brain-like AI that could run efficiently on neuromorphic hardware.
Limitations (acknowledged by the authors):
Experiments focused on relatively simple grayscale/binary images; scaling to complex natural images (e.g., CIFAR, high-resolution photos) remained an open challenge at the time.
Iterative inference adds per-sample compute overhead compared to pure feedforward backprop.
Sample quality and diversity do not yet match state-of-the-art implicit models like modern diffusion models or large GANs.
Still simplified compared to real cortical circuits (e.g., no spiking neurons, simplified neuron pairing).
Follow-up work by Ororbia and collaborators has since extended NGC to convolutional architectures (Conv-NGC) for natural images and even reinforcement learning settings, showing continued promise.
Why This Matters for the Future of AI
As AI systems grow larger and more energy-hungry, questions about biological plausibility, efficiency, and generalization become increasingly urgent. NGC demonstrates that we don't have to abandon performance to pursue more brain-like mechanisms. It shows that predictive coding principles can yield competitive generative models while offering advantages in sparsity, data efficiency, and multi-task utility. In a field still dominated by backpropagation, this work contributes to a growing "NeuroAI" movement exploring alternatives like equilibrium propagation, target propagation, and Hebbian learning. If scaled successfully, such frameworks could lead to AI that learns more like biological brains continuously, locally, and with far greater energy efficiency. The paper was selected as an Editor’s Choice shortly after publication and has influenced subsequent research in predictive coding for vision and control.
Conclusion
Ororbia and Kifer’s Neural Generative Coding framework is a thoughtful and empirically grounded step toward bridging neuroscience and machine learning. It doesn’t claim to replace backpropagation overnight, but it proves that biologically inspired alternatives can hold their own and sometimes outperform on key generative tasks.
As the authors conclude: even for complex probabilistic generative models, a more fundamentally brain-inspired approach to information processing and credit assignment can extract rich representations from data in an unsupervised fashion.