Affiliation

3rd Year, PhD
Computer Science
New York University
New York, NY, USA





Email:
zeiler@cs.nyu.edu















Deconvolutional Networks

Matthew D. Zeiler, Dilip Kirshnan, Graham W. Taylor, and Rob Fergus
Computer Vision and Pattern Recognition(June 13-18, 2010)

Abstract

Building robust low and mid-level image representations, beyond edge primitives, is a long-standing goal in vision. Many existing feature detectors spatially pool edge information which destroys cues such as edge intersections, parallelism and symmetry. We present a learning framework where features that capture these mid-level cues spontaneously emerge from image data. Our approach is based on the convolutional decomposition of images under a sparsity constraint and is totally unsupervised. By building a hierarchy of such decompositions we can learn rich feature sets that are a robust image representation for both the anal- ysis and synthesis of images.

Paper(.pdf)  Code  Images  Videos

Code

 Code for training a Deconvolutional Network and reconstructing images using a trained network is provided below. This can be used strictly with MATLAB only calls (much slower) or can use Intel IPP Convolutions Toolbox provided below to speed up the convolutions (requires Intel IPP Libraries).

A suggested package for fast LUT performance is Eero Simoncelli's MATLAB Pyramid Toolbox which can be found here: matlabPyrTools.
Also, for fast FFT-based convolutions (when not using IPP libraries) the Deconvoltional Network Toolbox now supports the followin implementation of n dimensional convolutions for matlab (which significantly increases non-IPP performance of this toolbox) Bruno Luong's convnfft.m.

Deconvolutional Network Toolbox

Author:
Matthew Zeiler (zeiler@cs.nyu.edu)
Date:
Apr 4, 2010
Version:
1.0.1 - A few bugs were fixed with plotting and saving figures. Also, the MATLABCentral convnfft.m file is now supported for enhanced performance during training (though not included with this package).
Instructions for Using this Toolbox:
Download and unzip this toolbox in the location of choice. To setup this toolbox, simply open matlab, cd to this directory, and type "setup" (without the quotes) in the command window. This sets up your path to include the required directories (you may want to add these paths into your startup.m file as well however) and opens complete documentation for this toolbox. Then you can modify parameters within /Defaults/set_defaults_here.m and run any of the scripts listed below (RUN THEM FROM THIS DIRECTORY as the results will be added to ./Results/ whereever you are running from).
This toolbox includes code that implements a Deconvolutional Network as described in the paper Deconvolutional Networks. This has a function to train a Deconvolutional Network, to visualize the learned filters, and to recsontruct a new image from a trained model. Also, there are files that can be used to make descriptors that can be used with Svetlana Lazebnik's Spatial Pyramid Matching code with a few minor modifications. The Deconvolutional Network Toolbox also works with the IPP Convolutions Toolbox which drastically improves performance (just ensure the IPP Convolutions Toolbox files are in your MATLAB path in order to use it with this toolbox). If you do not have IPP libraries, you can still get a significant performance enhancement by using convnfft.m from MATLABCentral.
Download (.zip)  Documentation (html)
Suggested Software

IPP Convolution Toolbox

Author:
Matthew Zeiler (zeiler@cs.nyu.edu)
Date:
Apr 4, 2010
Version:
1.0.0 - This is the initial release of this toolbox and may contain unkown bugs.
Instructions for Using this Toolbox:
Download and unzip this toolbox in the location of choice. To setup this toolbox, simply open matlab, cd to this directory, and type "setup" (without quotes) in the command window. This sets up your path to include the required directories (you may want to add these paths into your startup.m file as well however), tries to compile the IPP mex files included in this toolbox, and opens complete documentation for this toolbox. You will likely have to modify the paths in the /MEX/compilemex.m file to suite your machine's setup.
This toolbox includes Intel Performance Primitives (IPP) based convolutions of various types. The standard implementation ipp_conv2 can replace the MATLAB conv2 directly with significant speed-ups. These convolutions automatically switch to FFTs when the kernel size becomes large enough. Additionally, ipp_conv2 and the other included files have been extended beyond the functionality of MATLAB's conv2 as they can take in multiple images and multiple kernels. See their documentation for further details.
Download (.zip)  Documentation (html)

Images

 Most of the images are available in the pdf of the paper, however the following two images are included here as supplemental material. These represent the filters trained on a combination of the city and fruit datasets that were used to infer feature maps activation for the Caltech-101 recognition task.

Layer 1 Filters used for Caltech-101 experiments:
Layer 1 Filters used for Caltech-101 experiments.

Layer 2 Filters used for Caltech-101 experiments:
Layer 2 Filters used for Caltech-101 experiments.

Videos
The player will show in this paragraph unless you do not have flash player installed.

[Real-time learning of layer 1 filters on city dataset.]

[Real-time learning of layer 2 filters on city dataset.]

[Real-time learning of layer 1 filters on fruit dataset.]

[Real-time learning of layer 2 filters on fruit dataset.]

These videos are encoded using H.264 and so to view them with this integrated player you will need a relatively new version of Adobe Flash Player (called version 9 Update 3 or v9.0.115.0 which was released on December 3, 2007).