Bidirectional recurrent neural networks: Difference between revisions

Revision as of 21:58, 8 September 2022

Revision as of 16:10, 30 May 2023

Line 1:

{{short description|Type of artificial neural network}}

'''Bidirectional [[recurrent neural networks]]''' ('''BRNN''') connect two hidden layers of opposite directions to the same output. With this form of [[Generative model|generative deep learning]], the output layer can get information from past (backwards) and future (forward) states simultaneously. Invented in 1997 by Schuster and Paliwal,<ref name="Schuster">Schuster, Mike, and Kuldip K. Paliwal. "[https://www.researchgate.net/profile/Mike_Schuster/publication/3316656_Bidirectional_recurrent_neural_networks/links/56861d4008ae19758395f85c.pdf Bidirectional recurrent neural networks]." Signal Processing, IEEE Transactions on 45.11 (1997): 2673-2681.2. Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan</ref> BRNNs were introduced to increase the amount of input information available to the network. For example, [[multilayer perceptron]] (MLPs) and [[time delay neural network]] (TDNNs) have limitations on the input data flexibility, as they require their input data to be fixed. Standard [[recurrent neural network]] (RNNs) also have restrictions as the future input information cannot be reached from the current state. On the contrary, BRNNs do not require their input data to be fixed. Moreover, their future input information is reachable from the current state. <ref>{{Cite documentarxiv |title=Recent Advances in Recurrent Neural Networks|arxiv = 1801.01078|last1 = Salehinejad|first1 = Hojjat|last2 = Sankar|first2 = Sharan|last3 = Barfett|first3 = Joseph|last4 = Colak|first4 = Errol|last5 = Valaee|first5 = Shahrokh|year = 2017|bibcode = 2018arXiv180101078S}}</ref>

BRNN are especially useful when the context of the input is needed. For example, in [[handwriting recognition]], the performance can be enhanced by knowledge of the letters located before and after the current letter.

Line 8:

[[File:Structural diagrams of unidirectional and bidirectional recurrent neural networks.png|thumbnail|Structure of RNN and BRNN<ref name="Schuster" />|alt=|350x350px]]

The principle of BRNN is to split the neurons of a regular RNN into two directions, one for positive time direction (forward states), and another for negative time direction (backward states). Those two states’states' output are not connected to inputs of the opposite direction states. The general structure of RNN and BRNN can be depicted in the right diagram. By using two time directions, input information from the past and future of the current time frame can be used unlike standard RNN which requires the delays for including future information.<ref name="Schuster" />

==Training==