Graph neural network: Difference between revisions

Revision as of 14:51, 30 June 2022

Revision as of 05:59, 1 July 2022

Line 1:

{{Machine learning|Artificial neural network}}

A '''Graph neural network''' ('''GNN''') is a class of [[artificial neural network]]s for processing data that can be represented as [[Graph (abstract data type)|graphs]].<ref name="scarselli2008" /><ref name="sanchez2021" /><ref name="daigavane2021" />.

[[File:GNN building blocks.png|thumb|Basic building blocks of a Graph neural network (GNN). <math>(1)</math> Permutation equivariant layer. <math>(2)</math> Local pooling layer. <math>(3)</math> Global pooling (or readout) layer. Colors indicate [[Feature (machine learning)|features]].]]

In the more general subject of "Geometric [[Deep Learning]]", existing neural network architectures can be interpreted as GNNs operating on suitably defined graphs.<ref name=bronstein2021 />. [[Convolutional neural network]]s, in the context of [[computer vision]], can be seen as a GNN applied to graphs structured as grids of [[pixel]]s. [[Transformer (machine learning model) | Transformers]], in the context of [[natural language processing]], can be seen as GNNs applied to [[complete graph]]s whose nodes are [[words]] in a [[Sentence (linguistics) | sentence]].

The key design element of GNNs is the use of ''pairwise message passing'', such that graph nodes iteratively update their representations by exchanging information with their neighbors. Since their inception, several different GNN architectures have been proposed,<ref name="scarselli2008" /><ref name="kipf2016" /><ref name="hamilton2017" /><ref name="velickovic2018" />, which implement different flavors of message passing.<ref name=bronstein2021 />. {{As of|2022}}, whether is it possible to define GNN architectures "going beyond" message passing, or if every GNN can be built on message passing over suitably defined graphs, is an open research question.<ref name=velickovic2022 />.

Relevant application domains for GNNs include [[social network]]s,<ref name="ying2018" />,

[[Citation graph|citation networks]],<ref name="stanforddata" />,

[[molecular biology]],<ref name="gilmer2017" />,

[[physics]]<ref name=qasim2019 /> and

[[NP-hard]] [[combinatorial optimization]] problems.<ref name="li2018" />.

Several [[open source]] [[Library (computing)|libraries]] implementing Graph neural network are available, such as PyTorch Geometric<ref name=fey2019 /> ([[PyTorch]]), TensorFlow GNN<ref name=tfgnn2022 /> ([[TensorFlow]]), and jGraph<ref name=jraph2022/> ([[Google JAX]]).

Line 19:

== Architecture ==

The architecture of a generic GNN implements the following fundamental [[Layer (deep learning)|layers]]:<ref name=bronstein2021 />:

* <em>Permutation equivariant</em>: a permutation equivariant layer [[Map (mathematics)|maps]] a representation of a graph into an updated representation of the same graph. In literature, permutation equivariant layers are implemented via pairwise message passing between graph nodes.<ref name=bronstein2021 /><ref name=velickovic2022 />. Intuitively, in a message passing layer, nodes <em>update</em> their representations by <em>aggregating</em> the <em>messages</em> received from their immediate neighbours. As such, each message passing layer increases the receptive field of the GNN by one hop.

* <em>Local pooling</em>: a local pooling layer coarsens the graph via downsampling. Local pooling is used to increase the receptive field of a GNN, in a similar fashion to pooling layers in [[Convolutional neural network]]s. Examples include [[Nearest neighbor graph|k-nearest neighbours pooling]], top-k pooling,<ref name=gao2019 />, and self-attention pooling.<ref name=lee2019 />.

* <em>Global pooling</em>: a global pooling layer, also known as ''readout'' layer, provides fixed-size representation of the whole graph. The global pooling layer must be permutation invariant, such that permutations in the ordering of graph nodes and edges do not alter the final output.<ref name=lui2022 />. Examples include element-wise sum, mean or maximum.

It has been demonstrated that GNNs cannot be more expressive than the Weisfeiler–Lehman Graph Isomorphism Test.<ref name="douglas2011" /><ref name="xu2019" />. In practice, this means that there exist different graph structures (e.g., [[molecules]] with the same [[atoms]] but different [[Chemical bond|bonds]]) that cannot be distinguished by GNNs. More powerful GNNs operating on higher-dimension geometries such as [[Simplicial complex|simplicial complexes]] can be designed.<ref name=bronstein2021-2 />. {{As of|2022}}, whether or not future architectures will overcome the message passing primitive is an open research question.<ref name=velickovic2022 />.

[[File:GNN representational limits.png|thumb|[[Graph isomorphism|Non-isomorphic]] graphs that cannot be distinguished by a GNN due to the limitations of the Weisfeiler-Lehman Graph Isomorphism Test. Colors indicate node [[Feature (machine learning)|features]].]]

Line 33:

[[File:Message Passing Neural Network.png|thumb|Node representation update in a Message Passing Neural Network (MPNN) layer. Node <math>\mathbf{x}_0</math> receives messages sent by all of his immediate neighbours <math>\mathbf{x}_1</math> to <math>\mathbf{x}_4</math>. Messages are computing via the message function <math>\phi</math>, which accounts for the features of both senders and receiver.]]

Message passing layers are permutation-equivariant layers mapping a graph into an updated representation of the same graph. Formally, they can be expressed as message passing neural networks (MPNNs).<ref name="bronstein2021" />.

Let <math>G = (V,E)</math> be a [[Graph (discrete mathematics)|graph]], where <math>V</math> is the node set and <math>E</math> is the edge set. Let <math>N_u</math> [[Neighbourhood (graph theory)|neighbourhood]] of node <math>u \in V</math>. Additionally, let <math>\mathbf{x}_u</math> be the [[Feature (machine learning)|features]] of node <math>u \in V</math>, and <math>\mathbf{e}_{u, v}</math> be the features of edge <math>(u, v) \in E</math>. An MPNN [[Layer_(deep learning)|layer]] can be expressed as follows:<ref name="bronstein2021" />:

:<math>\mathbf{h}_u = \phi \left( \mathbf{x_u}, \bigoplus_{v \in N_u} \psi(\mathbf{x}_u, \mathbf{x}_v, \mathbf{e}_{uv}) \right)</math>

Where <math>\phi</math> and <math>\psi</math> are [[differentiable functions]] (e.g., [[artificial neural network]]s), and <math>\bigoplus</math> is a [[permutation]] [[Invariant (mathematics)|invariant]] aggregation operator that can accept an arbitrary number of inputs (e.g., element-wise sum, mean, or max). In particular, <math>\phi</math> and <math>\psi</math> are referred to as ''update'' and ''message'' functions, respectively. Intuitively, in an MPNN computational block, graph nodes ''update'' their representations by ''aggregating'' the ''messages'' received from their neighbours.

The outputs of one or more MPNN layers are node representations <math>\mathbf{h}_u</math> for each node <math>u \in V</math> in the graph. Node representations can be employed for any downstream task, such as node/graph [[Statistical classification|classification]] or edge prediction.

Graph nodes in an MPNN update their representation aggregating information from their immediate neighbours. As such, stacking <math>n</math> MPNN layers means that one node will be able to communicate with nodes that are at most <math>n</math> "hops" away. In principle, to ensure that every node receives information from every other node, one would need to stack a number of MPNN layers equal to the graph [[Distance (graph theory) | diameter]]. However, stacking many MPNN layers may cause issues such as oversmoothing<ref name=chen2021 /> and oversquashing.<ref name=alon2021 />. Oversmoothing refers to the issue of node representations becoming indistinguishable. Oversquashing refers to the bottleneck that is created by squeezing long-range dependencies into fixed-size representations.

Other "flavours" of MPNN have been developed in the literature,<ref name=bronstein2021 />, such as graph convolutional networks<ref name=kipf2016 /> and graph attention networks,<ref name=velickovic2018 />, whose definitions can be expressed in terms of the MPNN formalism.

=== Graph convolutional network ===

The graph convolutional network (GCN) was first introduced by [[Thomas Kipf]] and [[Max Welling]] in 2017.<ref name=kipf2016 />.

A GCN layer defines a [[Order of approximation|first-order approximation]] of a localized spectral [[Filter (signal processing)|filter]] on graphs. GCNs can be understood as a generalization of [[Convolutional Neural Network]]s to graph-structured data.

Line 60:

In particular, let <math>\mathbf{A}</math> be the graph adjacency matrix: then, one can define <math>\tilde{\mathbf{A}} = \mathbf{A} + \mathbf{I}</math> and <math>\tilde{\mathbf{D}_{ii}} = \sum_{j \in V} \tilde{A}_{ij}</math>, where <math>\mathbf{I}</math> denotes the [[identity matrix]]. This normalization ensures that the [[eigenvalue]]s of <math>\tilde{\mathbf{D}}^{-\frac{1}{2}} \tilde{\mathbf{A}} \tilde{\mathbf{D}}^{-\frac{1}{2}}</math> are bounded in the range <math>[0, 1]</math>, avoiding [[Numerical stability|numerical instabilities]] and [[Vanishing gradient problem|exploding/vanishing gradients]].

A limitation of GCNs is that they do not allow multidimensional edge features <math>\mathbf{e}_{uv}</math>.<ref name=kipf2016 />. It is however possible to associate scalar weights <math>w_{uv}</math> to each edge by imposing <math>A_{uv} = w_{uv}</math>, i.e., by setting each nonzero entry in the adjacency matrix equal to the weight of the corresponding edge.

=== Graph attention network ===

The graph attention network (GAT) was introduced by [[Petar Veličković]] et al. in 2018.<ref name=velickovic2018 />.

A multi-head GAT layer can be expressed as follows:

Line 86:

== Local pooling layers ==

Local pooling layers coarsen the graph via downsampling. In literature, several learnable local pooling strategies have been proposed.<ref name=lui2022 />.

=== Top-k pooling ===

Line 98:

:<math>\mathbf{A}' = \mathbf{A}_{\mathbf{i}, \mathbf{I}}</math>

Where <math>\mathbf{X}</math> is the matrix of node features, <math>\mathbf{A}</math> is the graph adjacency matrix, <math>\odot</math> denotes element-wise [[matrix multiplication]], and <math>\mathbf{p}</math> is a learnable [[Projection (mathematics)|projection]] vector. The projection vector <math>\mathbf{p}</math> computes a scalar projection value for each graph node. The nodes with the top-k highest projection scores are retained in the new adjacency matrix <math>\mathbf{A}'</math>. The <math>\text{sigmoid}(\cdot)</math> operation makes the projection vector <math>\mathbf{p}</math> trainable by [[backpropagation]], which otherwise would produce discrete outputs.<ref name=gao2019 />.

=== Self-attention pooling ===

Line 120:

{{See Also|AlphaFold}}

Graph neural networks are one of the main building blocks of [[AlphaFold]], an artificial intelligence program developed by [[Google]]'s [[DeepMind]] for solving the [[protein folding]] problem in [[biology]]. AlphaFold achieved first place in several [[CASP]] competitions.<ref name=guardian2018 /><ref name=mit2020 />.

=== Social Networks ===

{{See Also|Recommender system}}

[[Social networks]] are a major application ___domain for GNNs due to their natural representation as [[social graph]]s. GNNs are used to develop recommender systems based on both [[social relations]] and item relations.<ref name=fan2019 /><ref name=ying2018/>.

=== Combinatorial optimization ===

{{See Also|Combinatorial Optimization}}

GNNs are used as fundamental building blocks for several combinatorial optimization algorithm .<ref name=cappart2021 />. Examples include computing superior or competitive [[Placement (electronic design automation)|chip placements]] with handcrafted human solutions,<ref name=mirhoseini2021 />, and improving expert-designed bracing rules in [[Branch and Bound]].<ref name=gasse2019 />.

==References==

Line 134:

<ref name="scarselli2008">{{Cite journal|last1=Scarselli|first1=Franco|last2=Gori|first2=Marco|last3=Tsoi|first3=Ah Chung|last4=Hagenbuchner|first4=Markus|last5=Monfardini|first5=Gabriele|date=2009|title=The Graph Neural Network Model|url=https://ieeexplore.ieee.org/document/4700287|journal=IEEE Transactions on Neural Networks|volume=20|issue=1|pages=61–80|doi=10.1109/TNN.2008.2005605|pmid=19068426|s2cid=206756462|issn=1941-0093}}</ref>

<ref name="sanchez2021">{{Cite journal|last1=Sanchez-Lengeling|first1=Benjamin|last2=Reif|first2=Emily|last3=Pearce|first3=Adam|last4=Wiltschko|first4=Alex|date=2021-09-02|title=A Gentle Introduction to Graph Neural Networks|url=https://distill.pub/2021/gnn-intro|journal=Distill|language=en|volume=6|issue=9|pages=e33|doi=10.23915/distill.00033|issn=2476-0757|doi-access=free}}</ref>

<ref name="daigavane2021">{{Cite journal|last=Daigavane|first=Ameya|last2=Ravindran|first2=Balaraman|last3=Aggarwal|first3=Gaurav|date=2021-09-02|title=Understanding Convolutions on Graphs|url=https://distill.pub/2021/understanding-gnns|journal=Distill|language=en|volume=6|issue=9|pages=e32|doi=10.23915/distill.00032|issn=2476-0757}}</ref>

<ref name="gilmer2017">{{Cite journal|last1=Gilmer|first1=Justin|last2=Schoenholz|first2=Samuel S.|last3=Riley|first3=Patrick F.|last4=Vinyals|first4=Oriol|last5=Dahl|first5=George E.|date=2017-07-17|title=Neural Message Passing for Quantum Chemistry|url=http://proceedings.mlr.press/v70/gilmer17a.html|journal=International Conference on Machine Learning|language=en|publisher=PMLR|pages=1263–1272|arxiv=1704.01212}}</ref>

<ref name="kipf2016">{{Cite journal|last1=Kipf|first1=Thomas N|last2=Welling|first2=Max|date=2016|title=Semi-supervised classification with graph convolutional networks|url=https://ieeexplore.ieee.org/document/4700287|journal=International Conference on Learning Representations|volume=5|issue=1|pages=61–80|doi=10.1109/TNN.2008.2005605|pmid=19068426|arxiv=1609.02907|s2cid=206756462}}</ref>

Line 141:

<ref name=stanforddata>{{Cite web|title=Stanford Large Network Dataset Collection|url=https://snap.stanford.edu/data/|access-date=2021-07-05|website=snap.stanford.edu}}</ref>

<ref name="li2018">{{cite journal |last1=Li |first1=Zhuwen |last2=Chen |first2=Qifeng |last3=Koltun |first3=Vladlen |title=Combinatorial optimization with graph convolutional networks and guided tree search |journal=Neural Information Processing Systems |date=2018 |volume=31 |page=537--546 |url=https://arxiv.org/abs/1810.10659}}</ref>

<ref name="bronstein2021">{{cite journal |last1=Bronstein |first1=Michael M. |last2=Bruna |first2=Joan |last3=Cohen |first3=Taco |last4=Veličković |first4=Petar |title=Geometric Deep Learning: Grids, Groups, Graphs Geodesics and Gauges |journal=arXiv |date=May 4, 2021 |url=https://arxiv.org/abs/2104.13478}}</ref>.

<ref name=douglas2011>{{cite arXiv|last=Douglas|first=B. L.|date=2011-01-27|title=The Weisfeiler–Lehman Method and Graph Isomorphism Testing|class=math.CO|eprint=1101.5211}}</ref>.

<ref name=xu2019>{{Cite journal|last1=Xu|first1=Keyulu|last2=Hu|first2=Weihua|last3=Leskovec|first3=Jure|last4=Jegelka|first4=Stefanie|date=2019-02-22|title=How Powerful are Graph Neural Networks?|arxiv=1810.00826|journal=International Conference on Learning Representations|volume=7}}</ref>

<ref name=velickovic2022>{{cite journal |last1=Veličković |first1=Petar |title=Message passing all the way up |journal=arXiv |doi=10.48550/ARXIV.2202.11097 |url=https://arxiv.org/abs/2202.11097}}</ref>