Modeling pigeon behaviour using a Conditional Restricted Boltzmann Machine

Matthew D. Zeiler, Graham W. Taylor, Niko F. Troje, and Geoffrey E. Hinton

17th European Symposium on Artificial Neural Networks (April 22-24, 2009)
Supplemental Material:  Videos

Abstract
In an effort to better understand the complex courtship behaviour of pigeons, we have built a model learned from motion capture data. We employ a Conditional Restricted Boltzmann Machine (CRBM) with binary latent features and real-valued visible units. The units are conditioned on information from previous time steps to capture dynamics. We validate a trained model by quantifying the characteristic ‘head-bobbing’ present in pigeons. We also show how to predict missing data by marginalizing out the hidden variables and minimizing free energy.

Supplemental Materials

Videos

Single Pigeon Walking – Generated from a 2-layer Conditional RBM trained on pigeon data. Video is shown at 30 fps.

Pigeon “Head-Bobbing” Motion – Generated from a 2-layer Conditional RBM trained on pigeon data. Video is shown at 8 fps.

Pigeon “Head-Bobbing” Motion Overhead View – Generated from a 2-layer Conditional RBM trained on pigeon data. Grey lines represent the body coordinate frame with black lines being the head. Video is shown at 8 fps.

Pigeon “Left Foot Prediction” – 2-layer Conditional RBM used to predict the left foot (circles) using information from other body segments. Dots/lines indicate original data. Video is shown at 8 fps.

Pigeon “Motion Using Left Foot Prediction” – Playback using predictions from the above video combine with original data from other segments. Video is shown at 8 fps.

Pigeon “Right Foot Prediction” – 2-layer Conditional RBM used to predict the right foot (circles) using information from other body segments. Dots/lines indicate original data. Video is shown at 8 fps.

Pigeon “Motion Using Right Foot Prediction” – Playback using predictions from the above video combine with original data from other segments. Video is shown at 8 fps.

Pigeon “Two Foot Prediction” – 2-layer Conditional RBM used to predict the two foot (circles) using information from other body segments. Dots/lines indicate original data. Video is shown at 8 fps.

Pigeon “Motion Using Two Foot Prediction” – Playback using predictions from the above video combine with original data from other segments. Video is shown at 8 fps.

5-min Generated Walking Sequence – Generated from a 2-layer Conditional RBM trained on pigeon data. To be displayed to real pigeon to determine reaction. Video is shown at 30 fps.