Visualizing features, receptive fields, and classes in neural networks from "scratch" with Tensorflow 2. Part 2: Visualizing features and receptive fields

Summary¶

In the last section I talked about visualizing neural network classes by synthesizing images to maximally activate class labels. This time, we're going to do the same thing with convolutional filters rather than class labels. In addition, we'll play with a few other ways to look at the receptive fields of convolutional filters. All the figured here are generated from a single self-contained Jupyter notebook.

There are a few sections to this series:

1. Visualizing classes
2. Visualizing features
3. Reconstructing images from features
4. Transferring features between images

Synthesizing images that maximally activate convolutional filters / artificial neurons¶

We can use the same technique as above to synthesize images that maximally activate any neuron in the neural network. This will give us some idea about what that neuron is doing. Like the human visual stream, early neurons are responsive to simple textures and colors, while later neurons respond most strongly to complex patterns. As you move further into the network, the patterns have spatial structure that spans a larger range of the image. Below are a sample of a few images synthesized to maximally activate neurons at each layer of VGG 16. Note that for convolutional layers, these activations are maximized for each convolutional filter across the entire image, not just individual artifical neurons (we'll take a look at individual receptive fields in below).

Synthesizing images that maximally activate receptive fields of individual neurons¶

The previous example shows the maximum activation for a convolutional filter across a whole image. We could also pinpoint a single neuron (filter) and find an image that maximally activates that neuron. Because single neurons in CNNs are only receptive to a small region of the image (the receptive field), the synthesized image should only end up changing that region of the image. Here, we'll maximally activate single neurons to take a look at their receptive field. Deeper into the network receptive fields get larger, and more complex.

It might also be useful to take a look at those receptive fields, zoomed in.

Visualizing activation maps from neurons in different layers of a network¶

Another common way of visualizing features of a neural network is to take a look at what neurons are activated across an image for each filter type in a layer. Early in the network, there will be neurons that are specifically interested in colors or edges. Later, images respond to more complex features, sparsely present throughout the image.

All the code needed to reproduce these images and videos are available in self contained Jupyter notebooks.

In the next section, we'll try to see whether enough information is contained in the CNN filters to reconstruct the image at different layers. Thanks for reading.