Visualizing features, receptive fields, and classes in neural networks from "scratch" with Tensorflow 2. Part 2: Visualizing features and receptive fields

Posted on Tue 19 May 2020 in Neural networks


In the last section I talked about visualizing neural network classes by synthesizing images to maximally activate class labels. This time, we're going to do the same thing with convolutional filters rather than class labels. In addition, we'll play with a few other ways to look at the receptive fields of convolutional filters. All the figured here are generated from a single self-contained Jupyter notebook.

There are a few sections to this series:

  1. Visualizing classes Open In Colab
  2. Visualizing features Open In Colab
  3. Reconstructing images from features Open In Colab
  4. Transferring features between images Open In Colab

Synthesizing images that maximally activate convolutional filters / artificial neurons

We can use the same technique as above to synthesize images that maximally activate any neuron in the neural network. This will give us some idea about what that neuron is doing. Like the human visual stream, early neurons are responsive to simple textures and colors, while later neurons respond most strongly to complex patterns. As you move further into the network, the patterns have spatial structure that spans a larger range of the image. Below are a sample of a few images synthesized to maximally activate neurons at each layer of VGG 16. Note that for convolutional layers, these activations are maximized for each convolutional filter across the entire image, not just individual artifical neurons (we'll take a look at individual receptive fields in below).

Image synthesized to maximally activate filters from lower to higher layers in VGG16.

Synthesizing images that maximally activate receptive fields of individual neurons

The previous example shows the maximum activation for a convolutional filter across a whole image. We could also pinpoint a single neuron (filter) and find an image that maximally activates that neuron. Because single neurons in CNNs are only receptive to a small region of the image (the receptive field), the synthesized image should only end up changing that region of the image. Here, we'll maximally activate single neurons to take a look at their receptive field. Deeper into the network receptive fields get larger, and more complex.

Synthesized image to maximally activate a single artificial neuron with a receptive field in the center of the image.

It might also be useful to take a look at those receptive fields, zoomed in.

Synthesized image to maximally activate a single artificial neuron with a receptive field in the center of the image. Zoomed into recepetive field area.

Visualizing activation maps from neurons in different layers of a network

Another common way of visualizing features of a neural network is to take a look at what neurons are activated across an image for each filter type in a layer. Early in the network, there will be neurons that are specifically interested in colors or edges. Later, images respond to more complex features, sparsely present throughout the image.

Activation maps for convolutional filters in response to an image of Astronaut Eileen Collins.

All the code needed to reproduce these images and videos are available in self contained Jupyter notebooks.

In the next section, we'll try to see whether enough information is contained in the CNN filters to reconstruct the image at different layers. Thanks for reading.