The Sum Of Its Parts – Sensitivity In Neural Networks

Current frontier large language models (LLMs) are incomprehensibly large structures. Anthropic’s latest Fable 5 model is estimated to consist of approximately 5 trillion parameters and secures the top slot on the Artificial Analysis Intelligence benchmark. In many specialist test questions, the model outperforms human subject matter experts despite their computational advantage. The human brain roughly consists of 100 trillion synapses and runs on 20 watts of power. While parameter number and synapse count do not make for a good comparison due to their functional differences, this still gives an indication of the immense structures required to enable systems that resemble a thing we call intelligence.

The humungous size of a structure however is not a sufficient condition for intelligence. Intelligent systems are dynamic in that the primary intelligence emerges through the interaction of individual processing units. These neurons taken for themselves are unremarkable structures that can be easily understood by a human observer. The Rosenblatt Perceptron from 1957 is an example of such a structure and serves as a core concept for modern neural networks. The perceptron itself merely merges its inputs and is easily digested. Combine many neurons into a clever system and something akin to intelligence can be observed.

The idea that many trivial things combined are more than the sum of their parts is even older than Rosenblatt. In his foundational Metaphysics, estimated to be written long before Christ, Aristotle argues that

in the case of all things which have several parts and in which the totality is not, as it were, a mere heap, but the whole is something beside the parts, there is a cause.

The combination of several parts into one unit thus yields more than we might initially estimate simply summing up said parts. In complex systems theory, this behaviour is called the emergence of a property. From the interactions between trivial pieces emerges a behaviour that is more than the sum of its parts. Nature’s best example for such a kind of system is an ant colony. Despite each ant having limited knowledge and capability, the interaction between individual ants guarantees that an ant-bed is built. In this essay series, I’m setting out to explore if and if yes, how neural networks exhibit complexity and emergence phenomena.

To start this journey, I trained a small convolutional neural network (CNN) on the MNIST dataset containing handwritten numbers from 0 to 9. The architecture of the CNN roughly follows the LeNet from 1998, one of the simplest networks performing well on the MNIST task. As an educational exercise, all parts of this network are hand-coded using only numpy. Both performance and capability thus lag far behind current frontier models or more modern CNNs. This network implementation achieved a test accuracy of 98.9% after 11 epochs.

Before training the network on MNIST data, each layer must be assigned a starting weight. Initial weight choice is a surprisingly important parameter as stacked layers with misaligned initial weight scales can lead to vanishing (an ordered phase of the system) or exploding gradients during training (a chaotic phase of the system). For all weights, we use the He initialisation protocol where the weights of each layer $W_{i}$ are drawn from the normal distribution

$$W_{i} \propto \mathcal{N}\left(0, \frac{\sigma_w^2}{L_i}\right)$$

with the width of each layer given as $L_i$. He initialisation aims to begin the training process at the critical point, that is the point where the network sits at the edge between ordered and chaotic phases. This initialisation can be easily demonstrated by sweeping a simple $50$ layer ReLU stack for different values of $\sigma_w^2$. We set $\sigma_w^2 = 2\alpha$ and sweep $\alpha$ across a range of values while measuring the mean activation norm $q^l$. That is, we run $50$ random inputs through the same weights up to layer $l$ and then take the average of those $50$ numbers. This yields a narrow region around $\alpha = 1$ for which the activation norm stays flat across all layers. For $\alpha <1$ we observe an ordered phase of the network with convergence towards $0$ while for $\alpha > 1$ the chaotic phase becomes dominant with values diverging. We can therefore conclude that this simplified network does indeed start training at a critical point. What remains open is if that claim transfers to our LeNet-like network and if it does, what happens to that critical point during training.

An intuitive way to think about that question is to consider the network as one function from the input to the output space. In this MNIST example, the network maps $\mathbb{R}^{784} \to \mathbb{R}^{10}$, that is we map all input pixels of the picture of a number to the ten number output categories $0-9$ as real-valued scores. This map can be represented its Jacobian $J \in \mathbb{R}^{10 \times 784}$, the matrix of all partial derivatives of the output relative to the input. We can use $J$ to investigate how sensitive the network is in the direction of the input and whether there exists something like a preferential direction.

An immediate way to look at $J$ is to plot it for each original $28\times28$ shape and look at the then resulting images that match the output categories. Here, red indicates high sensitivity to that input pixel and blue shows dampening of these input directions. The following image shows the rescaled Jacobian for an input image that contains a $7$.

If the network has learned what makes say a $7$ a $7$, we would expect a $7$-like pattern light up in the Jacobian of that digit, with all other pixels being suppressed. A class that can be easily confused (in this case a $1$) should show strong blue patterning across the input. We do not expect the network to be highly sensitive towards $1$-nes if the input in fact is a $7$. From the plot above, we see that no clear sensitivity pattern emerges. $5$, $7$, and $9$ output category Jacobians are more saturated than other classes, indicating that the network is especially sensitive for these. Note that large parts of the Jacobian are neutral colour – these input pixels are practically dead as the network show no sensitivity to them whatsoever.

With this approach, we can say how the output reacts to each individual pixel in the input image. We find that there is no immediately obvious sensitivity pattern emerging for each output class. The more interesting question then is if we can find some combination of pixels that the network is particularly sensitive or blind to. In other words: over every possible direction of perturbation in the input, which one moves the ouput the most and the least?

Luckily, $J$ can produce a direct answer. Any linear map morphs a sphere on the unit circle into an ellipsoid. We can decompose that morph into three components: $V^T$ notes wich directions of the input become the ellipsoid axes (a rotation), $\Sigma$ describes how long the resulting axes are (a stretch), and $U$ represents in which directions those axes point in the output space (another rotation). For our Jacobian $J \in \mathbb{R}^{10\times784}$ this means that $784$-dimensional unit sphere goes in and only $10$ dimensions come out of it.

$$J = U \Sigma V^T$$

This process is called singular value decomposition and by applying it to our network, we find $10$ orthogonal input directions $v_1, ..., v_{10}$ (columns of $V \in \mathbb{R}^{784 \times10}$), $10$ orthogonal output directions $u_1, ..., u_{10}$ (columns of $U \in \mathbb{R}^{10\times10}$), and scaling factors $s_1 \geq ... \geq s_{10} \geq 0$, the diagonal of $\Sigma \in \mathbb{R}^{10\times784}$.

Nudging the input along $v_i$ moves the output along $u_i$, scaled by $s_i$. A $s_i$ value of $1$ means that input direction propagates through the input unchanged, whereas a value smaller $1$ indicates damping and greater $1$ indicates amplification. We then compute this decomposition over many different input images during training and take the sample mean of $\Sigma$ for each training epoch. In the image below, we plot the averaged $s_i$ values for each singular value direction and plot the scaling factor progression over training epochs.

At the beginning of the training, the spectrum of the singular values sits close to $s=1$. This is what the He initialisation setting of $\sigma_w^2=2$ predicts. During training, each of the SVD directions moves at a very different pace. While $s_1$ has grown sevenfold from $2.5$ to roughly $18$, $s_{10}$ has barely moved relatively. The gap between the least and the most sensitive input direction widens during training. So much so that the upper SVD indices $9$ and $10$ barely contribute to network sensitivity. The effective rank of the Jacobian $J$ collapses to around $5$ after the first two epochs.

The technical part of this essay started with two questions: would He initialisation at criticality transfer to a more complicated structure such as our LeNet-like MNIST classifier implementation? And if it did, what would happen to that criticality during training? Using singular value decomposition, we have shown that only part of the SVD spectrum sits around criticality at $s=1$ after initialisation. The most sensitive direction $s_1 \approx 2.4$ is pretty far away from that value. He initialisation therefore only partly translates to networks that contain layers other than fully connected and ReLU. During training, we observed that the most sensitive directions with low SVD indices grow quicker per epoch than the less sensitive ones. The network thus moves away rapidily from the weak criticality given by the He initialisiation. During training, the network quickly concentrates on the energy equivalent of five preferred input directions out of the ten available and begins to optimise on these.

Did we learn something about the existence of emergence in neural networks by studying the critical point at initialisation and during training through the lens of the Jacobian? Not really. The assumption detecting a critical point by applying a narrow mathematical instrument was perhaps too naive. And yet, more questions arise from this experiment. Why does the network focuses on these specific input directions? How does this selection emerge? And what happens between the input and output layers? In the next installment of this series, we will look at network weights and what they can tell us.