Deconvolutional Networks for Feature Learning

Matthew D. Zeiler, Dilip Krishnan, Graham W. Taylor, and Rob Fergus
The Learning (Snowbird) Workshop (April 6-9, 2010)

Abstract

Building robust low-level image representations, beyond edge primitives, is a long-standing goal in vision. In its most ba- sic form, an image is a matrix of intensities. How we should progress from this matrix to stable mid-level representations, useful for high-level vision tasks, remains unclear. Popular feature representations such as SIFT or HOG spatially pool edge information to form descriptors that are invariant to local transformations. However, in doing so important cues such as edge intersections, grouping, parallelism and symmetry are lost...

Paper(.pdf)  Code  Images  Videos

Code

Code for training a Deconvolutional Network and reconstructing images using a trained network is provided below. This can be used strictly with MATLAB only calls (much slower) or can use Intel IPP Convolutions Toolbox provided below to speed up the convolutions (requires Intel IPP Libraries).

A suggested package for fast LUT performance is Eero Simoncelli's MATLAB Pyramid Toolbox which can be found here: matlabPyrTools.
Also, for fast FFT-based convolutions (when not using IPP libraries) the Deconvoltional Network Toolbox now supports the followin implementation of n dimensional convolutions for matlab (which significantly increases non-IPP performance of this toolbox) Bruno Luong's convnfft.m.

Deconvolutional Network Toolbox

Author:
Matthew Zeiler ([email protected])
Date:
Apr 4, 2010
Version:
1.0.1 - A few bugs were fixed with plotting and saving figures. Also, the MATLABCentral convnfft.m file is now supported for enhanced performance during training (though not included with this package).
Note: The Adaptive Deconvolutional Network Toolbox can do everything that this older code could do and I recommend using that toolbox as it is much simpler.
Instructions for Using this Toolbox:
Download and unzip this toolbox in the location of choice. To setup this toolbox, simply open matlab, cd to this directory, and type "setup" (without the quotes) in the command window. This sets up your path to include the required directories (you may want to add these paths into your startup.m file as well however) and opens complete documentation for this toolbox. Then you can modify parameters within /Defaults/set_defaults_here.m and run any of the scripts listed below (RUN THEM FROM THIS DIRECTORY as the results will be added to ./Results/ whereever you are running from).
This toolbox includes code that implements a Deconvolutional Network as described in the paper Deconvolutional Networks. This has a function to train a Deconvolutional Network, to visualize the learned filters, and to recsontruct a new image from a trained model. Also, there are files that can be used to make descriptors that can be used with Svetlana Lazebnik's Spatial Pyramid Matching code with a few minor modifications. The Deconvolutional Network Toolbox also works with the IPP Convolutions Toolbox which drastically improves performance (just ensure the IPP Convolutions Toolbox files are in your MATLAB path in order to use it with this toolbox). If you do not have IPP libraries, you can still get a significant performance enhancement by using convnfft.m from MATLABCentral.

Note: I highly recommend the Adaptive Deconvolutional Network Toolbox rather than this one as it is easier to use and has more features.
Download (.zip)  Documentation (html)
Suggested Software

IPP Convolution Toolbox

Author:
Matthew Zeiler ([email protected])
Date:
Apr 4, 2010
Version:
1.0.0 - This is the initial release of this toolbox and may contain unkown bugs.
Instructions for Using this Toolbox:
Download and unzip this toolbox in the location of choice. To setup this toolbox, simply open matlab, cd to this directory, and type "setup" (without quotes) in the command window. This sets up your path to include the required directories (you may want to add these paths into your startup.m file as well however), tries to compile the IPP mex files included in this toolbox, and opens complete documentation for this toolbox. You will likely have to modify the paths in the /MEX/compilemex.m file to suite your machine's setup.
This toolbox includes Intel Performance Primitives (IPP) based convolutions of various types. The standard implementation ipp_conv2 can replace the MATLAB conv2 directly with significant speed-ups. These convolutions automatically switch to FFTs when the kernel size becomes large enough. Additionally, ipp_conv2 and the other included files have been extended beyond the functionality of MATLAB's conv2 as they can take in multiple images and multiple kernels. See their documentation for further details.
Download (.zip)  Documentation (html)


Images

Most of the images are available in the pdf of the paper, however the following two images are included here as supplemental material. These represent the filters trained on a combination of the city and fruit datasets that were used to infer feature maps activation for the Caltech-101 recognition task.

Layer 1 Filters used for Caltech-101 experiments:
Layer 1 Filters used for Caltech-101 experiments.

Layer 2 Filters used for Caltech-101 experiments:
Layer 2 Filters used for Caltech-101 experiments.


Videos

The player will show in this paragraph unless you do not have flash player installed.

[Real-time learning of layer 1 filters on city dataset.]

[Real-time learning of layer 2 filters on city dataset.]

[Real-time learning of layer 1 filters on fruit dataset.]

[Real-time learning of layer 2 filters on fruit dataset.]

These videos are encoded using H.264 and so to view them with this integrated player you will need a relatively new version of Adobe Flash Player (called version 9 Update 3 or v9.0.115.0 which was released on December 3, 2007).