Time delay neural network: Difference between revisions

Revision as of 18:31, 10 March 2019

Revision as of 15:03, 13 June 2019

Line 13:

The Time Delay Neural Network, like other neural networks, operates with multiple interconnected layers of [[perceptron]]s, and is implemented as a [[feedforward neural network]]. All neurons (at each layer) of a TDNN receive inputs from the outputs of neurons at the layer below but with two differences:

# Unlike regular [[Multilayer perceptron|Multi-Layer perceptrons]], all units in a TDNN, at each layer, obtain inputs from a contextual ''window'' of outputs from the layer below. For time varying signals (e.g. speech), each unit has connections to the output from units below but also to the time-delayed (past) outputs from these same units. This models the units' temporal pattern/trajectory. For two-dimensional signals (e.g. time-frequency patterns or images), a 2-D context window is observed at each layer. Higher layers have inputs from wideningwider context windows than lower layers and thus generally model coarser levels of abstraction.

# Shift-invariance is achieved by explicitly removing position dependence during [[backpropagation]] training. This is done by making time-shifted copies of a network across the dimension of invariance (here: time). The error gradient is then computed by backpropagation through all these networks from an overall target vector, but before performing the weight update, the error gradients associated with shifted copies are averaged and thus shared and constraint to be equal. Thus, all position dependence from backpropagation training through the shifted copies is removed and the copied networks learn the most salient hidden features shift-invariantly, i.e. independent of their precise position in the input data. Shift-invariance is also readily extended to multiple dimensions by imposing similar weight-sharing across copies that are shifted along multiple dimensions.<ref name=":1" /><ref name=":2" />