Structure tensor: Difference between revisions

Revision as of 14:27, 3 January 2021

Latest revision as of 19:10, 23 May 2025

(33 intermediate revisions by 18 users not shown)

Line 1:

{{FeatureDetectionCompVisNavbox}}{{Use American English|date = March 2019}}

{{Short description|Tensor related to gradients}}

{{FeatureDetectionCompVisNavbox}}

In mathematics, the '''structure [[tensor]]''', also referred to as the '''second-moment matrix''', is a [[matrix (mathematics)|matrix]] derived from the [[gradient]] of a [[function (mathematics)|function]]. It summarizes the predominant directions of the gradient in a specified neighborhood of a point, and the degree to which those directions are coherent. The structure tensor is often used in [[image processing]] and [[computer vision]].<ref name=bigun86>

{{FeatureDetectionCompVisNavbox}}{{Use American English|date = March 2019}}

J. Bigun and G. Granlund (1986), ''Optimal Orientation Detection of Linear Symmetry''. Tech. Report LiTH-ISY-I-0828, Computer Vision Laboratory, Linkoping University, Sweden 1986; Thesis Report, Linkoping studies in science and technology No. 85, 1986.

In mathematics, the '''structure [[tensor]]''', also referred to as the '''second-moment matrix''', is a [[matrix (mathematics)|matrix]] derived from the [[gradient]] of a [[function (mathematics)|function]]. It summarizesdescribes the predominant directionsdistribution of the gradient in a specified neighborhood ofaround a point, and makes the degreeinformation invariant to whichthe thoseobserving coordinates<!-- Example: if you have a 2D image with two components storing the gradient direction and a Gaussian blur is performed separately on each component, the result will be ill-formed (specially for the directions arewere coherentvector orientations flip). On the other hand if the blur is performed component-wise on a 2x2 structure tensor the main eigenvector (scaled by its eigenvalue) will properly represent the gradient. -->. The structure tensor is often used in [[image processing]] and [[computer vision]].<ref name=bigun86>

J. Bigun and G. Granlund (1986), ''Optimal Orientation Detection of Linear Symmetry''. Tech. Report LiTH-ISY-I-0828, Computer Vision Laboratory, Linkoping University, Sweden 1986; Thesis Report, Linkoping studies in science and technology No. 85, 1986.

</ref><ref name=bigun87>

{{cite conference|author1=J. Bigun |author2=G. Granlund |name-list-style=amp |titlechapter=Optimal Orientation Detection of Linear Symmetry|___location=Piscataway|booktitletitle=First int. Conf. on Computer Vision, ICCV, (London) |publisher=IEEE Computer Society Press, Piscataway|pages=433–438|year=1987 }}

</ref><ref name=knutsson89>

{{cite conference|author=H. Knutsson|titlechapter=Representing local structure using tensors|___location=Oulu|booktitletitle=Proceedings 6th Scandinavian Conf. on Image Analysis|publisher=Oulu University|pages=244–251|year=1989}}

</ref>

Line 12 ⟶ 13:

===Continuous version===

For a function <math>I</math> of two variables {{nowrapmath|1=''p'' = (''x'', ''y'')}}, the structure tensor is the 2×2 matrix

:<math display="block">

S_w(p) =

\begin{bmatrix}

\int w(r) (I_x(p-r))^2\,d rdr & \int w(r) I_x(p-r)I_y(p-r)\,d rdr \\[10pt]

\int w(r) I_x(p-r)I_y(p-r)\,d r dr & \int w(r) (I_y(p-r))^2\,d rdr

\end{bmatrix}

</math>

where <math>I_x</math> and <math>I_y</math> are the [[partial derivative]]s of <math>I</math> with respect to ''x'' and ''y''; the integrals range over the plane <math>\mathbb{R}^2</math>; and ''w'' is some fixed "window function" (such as a [[Gaussian blur]]), a [[distribution (mathematics)|distribution]] on two variables. Note that the matrix <math>S_w</math> is itself a function of {{nowrapmath|1=''p'' = (''x'', ''y'')}}.

The formula above can be written also as <math display="inline">S_w(p) = \int w(r) S_0(p-r) \,d rdr</math>, where <math>S_0</math> is the matrix-valued function defined by

:<math display="block">

S_0(p)=

\begin{bmatrix}

Line 32 ⟶ 33:

</math>

If the [[gradient]] <math>\nabla I = (I_x,I_y)^\text{T}</math> of <math>I</math> is viewed as a 2×1 (single-column) matrix, where <math>(.\cdot)^\text{T}</math> denotes [[transpose]] operation, turning a row vector to a column vector, the matrix <math>S_0</math> can be written as the [[matrix product]] <math>(\nabla I)(\nabla I)^\text{T}</math>, alsoor known[[tensor asproduct|tensor anor outer product,]] or<math>\nabla tensorI product.\otimes \nabla I</math>. Note however that the structure tensor <math>S_w(p)</math> cannot be factored in this way in general except if <math>w</math> is a [[Dirac delta function]].

===Discrete version===

In image processing and other similar applications, the function <math>I</math> is usually given as a discrete [[array data structure|array]] of samples <math>I[p]</math>, where ''p'' is a pair of integer indices. The 2D structure tensor at a given [[pixel]] is usually taken to be the discrete sum

:<math display="block">

S_w[p] =

\begin{bmatrix}

\sum_r w[r] (I_x[p-r])^2 & \sum_r w[r] I_x[p-r]I_y[p-r] \\[10pt]

\sum_r w[r] I_x[p-r]I_y[p-r] & \sum_r w[r] (I_y[p-r])^2

\end{bmatrix}

</math>

Here the summation index ''r'' ranges over a finite set of index pairs (the "window", typically <math>\{-m.. \ldots +m\}\times\{-m.. \ldots +m\}</math> for some ''m''), and ''w''[''r''] is a fixed "window weight" that depends on ''r'', such that the sum of all weights is 1. The values <math>I_x[p],I_y[p]</math> are the partial derivatives sampled at pixel ''p''; which, for instance, may be estimated from by <math>I</math> by [[finite difference]] formulas.

The formula of the structure tensor can be written also as <math display="inline">S_w[p]=\sum_r w[r] S_0[p-r]</math>, where <math>S_0</math> is the matrix-valued array such that

:<math display="block">

S_0[p] =

\begin{bmatrix}

(I_x[p])^2 & I_x[p]I_y[p] \\[10pt]

I_x[p]I_y[p] & (I_y[p])^2

\end{bmatrix}

</math>

Line 59 ⟶ 60:

The importance of the 2D structure tensor <math>S_w</math> stems from the fact [[eigenvalue]]s <math>\lambda_1,\lambda_2</math> (which can be ordered so that <math>\lambda_1 \geq \lambda_2\geq 0</math>) and the corresponding [[eigenvector]]s <math>e_1,e_2</math> summarize the distribution of the [[gradient]] <math>\nabla I = (I_x,I_y)</math> of <math>I</math> within the window defined by <math>w</math> centered at <math>p</math>.<ref name=bigun86/><ref name=bigun87/><ref name=knutsson89/>

Namely, if <math>\lambda_1 > \lambda_2</math>, then <math>e_1</math> (or <math>-e_1</math>) is the direction that is maximally aligned with the gradient within the window.

In particular, if <math>\lambda_1 > 0, \lambda_2 = 0</math> then the gradient is always a multiple of <math>e_1</math> (positive, negative or zero); this is the case if and only if <math>I</math> within the window varies along the direction <math>e_1</math> but is constant along <math>e_2</math>. This condition of eigenvalues is also called linear symmetry condition because then the iso-curves of <math>I</math> consist in parallel lines, i.e there exists a one dimensional function <math>g </math> which can generate the two dimensional function <math>I</math> as <math>I(x,y)=g(d^\text{T} p)</math> for some constant vector <math>d=(d_x,d_y)^T </math> and the coordinates <math>p=(x,y)^T </math>.

If <math>\lambda_1 = \lambda_2</math>, on the other hand, the gradient in the window has no predominant direction; which happens, for instance, when the image has [[rotational symmetry]] within that window. This condition of eigenvalues is also called balanced body, or directional equilibrium condition because it holds when all gradient directions in the window are equally frequent/probable.

Furthermore, the condition <math>\lambda_1 = \lambda_2 = 0</math> happens if and only if the function <math>I</math> is constant (<math>\nabla I = (0,0)</math>) within <math>W</math>.

More generally, the value of <math>\lambda_k </math>, for ''k''=1 or ''k''=2, is the <math>w</math>-weighted average, in the neighborhood of ''p'', of the square of the [[directional derivative]] of <math>I</math> along <math>e_k</math>. The relative discrepancy between the two eigenvalues of <math>S_w</math> is an indicator of the degree of [[isotropy|anisotropy]] of the gradient in the window, namely how strongly is it biased towards a particular direction (and its opposite).<ref name="Jahne1993">

{{cite book|author=B. Jahne|title=Spatio-Temporal Image Processing: Theory and Scientific Applications| ___location=Berlin| publisher=Springer-Verlag| volume=751|year=1993}}

</ref><ref name=MedioniEA>

{{cite book|author1=G. Medioni, M. Lee |author2=C. Tang |name-list-style=amp |title=A Computational Framework for Feature Extraction and Segmentation|publisher=Elsevier Science|date=March 2000}}

</ref> This attribute can be quantified by the '''coherence''', defined as

:<math display="block">c_w=\left(\frac{\lambda_1-\lambda_2}{\lambda_1+\lambda_2}\right)^2</math>

if <math>\lambda_2>0</math>. This quantity is 1 when the gradient is totally aligned, and 0 when it has no preferred direction. The formula is undefined, even in the [[limit of a function|limit]], when the image is constant in the window (<math>\lambda_1=\lambda_2=0</math>). Some authors define it as 0 in that case.

Note that the average of the gradient <math>\nabla I</math> inside the window is '''not''' a good indicator of anisotropy. Aligned but oppositely oriented gradient vectors would cancel out in this average, whereas in the structure tensor they are properly added together.<ref>

{{cite journaltech report|authorauthor1=T. Brox, |author2= J. Weickert, |author3=B. Burgeth and |author4=P. Mrazek |name-list-style=amp |title=Nonlinear Structure Tensors|booktitleinstitution=Universität des Saarlandes, Tech. Report|issuenumber=113|pages=1–32|year=2004}}

</ref> This is a reason for why <math>(\nabla I)(\nabla I)^\text{T}</math> is used in the averaging of the structure tensor to optimize the direction instead of <math>\nabla I</math>.

By expanding the effective radius of the window function <math>w</math> (that is, increasing its variance), one can make the structure tensor more robust in the face of noise, at the cost of diminished spatial resolution.<ref name=MedioniEA /><ref name=lin94book>T. Lindeberg (19941993), ''[http://www.nadacsc.kth.se/~tony/book.html Scale-Space Theory in Computer Vision]''. Kluwer Academic Publishers, (see sections 14.4.1 and 14.2.3 on pages 359–360 and 355–356 for detailed statements about how the multi-scale second-moment matrix/structure tensor defines a true and uniquely determined multi-scale representation of directional data).

</ref> The formal basis for this property is described in more detail below, where it is shown that a multi-scale formulation of the structure tensor, referred to as the [[Structure tensor#The multi-scale structure tensor|multi-scale structure tensor]], constitutes a ''true multi-scale representation of directional data under variations of the spatial extent of the window function''.

===Complex version===

The interpretation and implementation of the 2D structure tensor becomes particularly accessible using [[complex numbersnumber]]s.<ref name="bigun87" /> The structure tensor consists in 3 real numbers

:<math display="block">

S_w(p) =

\begin{bmatrix}

\mu_{20} & \mu_{11} \\[10pt]

\mu_{11} & \mu_{02}

\end{bmatrix}

</math>

where <math display="inline">

\mu_{20} =\int (w(r) (I_x(p-r))^2\,d r dr

</math> , <math display="inline">

\mu_{02} =\int (w(r) (I_y(p-r))^2\,d r dr

</math> and <math display="inline">

\mu_{11} =\int w(r) I_x(p-r)I_y(p-r)\,d r dr

</math> in which integrals can be replaced by summations for discrete representation. Using [[Parseval's relationshipidentity]] it is clear that the three real numbers are the second order moments of the power spectrum of <math>I</math>. The following second order complex moment of the power spectrum of <math>I</math> can then be written as

<math display="inlineblock">

\kappa_{20} = \mu_{20}-\mu_{02}+i2\mu_{11}=\int (w(r) (I_x(p-r)+i I_y(p-r))^2 \,d r dr =(\lambda_1-\lambda_2)\exp(i2\phi)

</math>

where <math>i=\sqrt{-1}</math> and <math>\phi</math> is the direction angle of the most significant eigenvector of the structure tensor <math>\phi=\angle{e_1}</math> whereas <math>\lambda_1</math> and <math>\lambda_2</math> are the most and the least significant eigenvalues. From, this it follows that <math>\kappa_{20} </math> contains both a certainty <math>|\kappa_{20}|=\lambda_1-\lambda_2</math> and the optimal direction in double angle representation since it is a complex number consisting of two real numbers. It follows also that if the gradient is represented as a complex number, and is remapped by squaring (i.e. the argument angles of the complex gradient is doubled), then averaging acts as an optimizer in the mapped ___domain, since it directly delivers both the optimal direction (in double angle representation) and the associated certainty. The complex number represents thus how much linear structure (linear symmetry) there is in image <math>I</math>, and the complex number is obtained directly by averaging the gradient in its (complex) double angle representation without computing the eigenvalues and the eigenvectors explicitly.

Likewise the following second order complex moment of the power spectrum of <math>I</math>, which happens to be always real because <math>I</math> is real,

<math display="inlineblock">

\kappa_{11} = \mu_{20}+\mu_{02} = \int (w(r) |I_x(p-r)+i I_y(p-r)|^2 \,d rdr = =\lambda_1+\lambda_2

</math>

can be obtained, with <math>\lambda_1</math> and <math>\lambda_2</math> being the eigenvalues as before. Notice that this time the magnitude of the complex gradient is squared (which is always real).

However, decomposing the structure tensor in its eigenvectors yields its tensor components as

:<math display="block">

S_w(p) =

\lambda_1 e_1e_1^\text{T}+

\lambda_2 e_2e_2^\text{T} =(\lambda_1 -\lambda_2)e_1e_1^\text{T}+

\lambda_2( e_1e_1^\text{T}+e_2e_2^\text{T})= (\lambda_1 -\lambda_2)e_1e_1^\text{T}+

\lambda_2 E

</math>

where <math>E</math> is the identity matrix in 2D because the two eigenvectors are always orthogonal (and sum to unity). The first term in the last expression of the decomposition, <math>(\lambda_1 -\lambda_2)e_1e_1^\text{T}</math>, represents the linear symmetry component of the structure tensor containing all directional information (as a rank-1 matrix), whereas the second term represents the balanced body component of the tensor, which lacks any directional information (containing an identity matrix <math>E</math>). To know how much directional information there is in <math>I</math> is then the same as checking how large <math>\lambda_1-\lambda_2 </math> is compared to <math>\lambda_2</math>.

Evidently, <math>\kappa_{20}</math> is the complex equivalent of the first term in the tensor decomposition, whereas <math display="block">\tfrac 1 2 (|\kappa_{20}|-\kappa_{11})/2 = \lambda_2</math>is the equivalent of the second term. Thus the two scalars, comprising three real numbers,

:<math display="block">

\begin{array}{calign}

\kappa_{20} &=& (\lambda_1-\lambda_2)\exp(i2\phi) &=& w*(h*I)^2 \\

\kappa_{11} &=& \lambda_1+\lambda_2&=&w*|h*I|^2 \\

\end{arrayalign}

</math>

where <math>h(x,y)=(x+iy)\exp(-(x^2+y^2)/(2\sigma^2)) </math> is the (complex) gradient filter, and <math>*</math> is convolution, constitute a complex representation of the 2D Structure Tensor. As discussed here and elsewhere <math>w</math> defines the local image which is usually a Gaussian (with a certain variance defining the outer scale), and <math>\sigma </math> is the (inner scale) parameter determining the effective frequency range in which the orientation <math>2\phi</math> is to be estimated.

The elegance of the complex representation stems from that the two components of the structure tensor can be obtained as averages and independently. In turn, this means that <math>\kappa_{20}</math> and <math>\kappa_{11}</math> can be used in a scale space representation to describe the evidence for presence of unique orientation and the evidence for the alternative hypothesis, the presence of multiple balanced orientations, without computing the eigenvectors and eigenvalues. A functional, such as squaring the complex numbers have to this date not been shown to exist for structure tensors with dimensions higher than two. In Bigun 91, it has been put forward with due argument that this is because complex numbers are commutative algebras whereas quaternions, the possible candidate to construct such a functional by, constitute a non-commutative algebra.<ref name=bigun91>

{{cite journal|author1=J. Bigun |author2=G. Granlund |author3=J. Wiklund |name-list-style=amp |title=Multidimensional Orientation Estimation with Applications to Texture Analysis and Optical Flow| journal= IEEE Transactions on Pattern Analysis and Machine Intelligence|volume=13|number=8|pages=775–790|year=1991 |doi=10.1109/34.85668}}</ref>

The complex representation of the structure tensor is frequently used in fingerprint analysis to obtain direction maps containing certainties which in turn are used to enhance them, to find the locations of the global (cores and deltas) and local (minutia) singularities, as well as automatically evaluate the quality of the fingerprints.

==The 3D structure tensor==

===Definition===

The structure tensor can be defined also for a function <math>I</math> of three variables ''p''=(''x'',''y'',''z'') in an entirely analogous way. Namely, in the continuous version we have <math display="inline">S_w(p) = \int w(r) S_0(p-r) \,d rdr</math>, where

:<math display="block">

S_0(p) =

\begin{bmatrix}

(I_x(p))^2 & I_x(p)I_y(p) & I_x(p)I_z(p) \\[10pt]

Line 158 ⟶ 159:

where <math>I_x,I_y,I_z</math> are the three partial derivatives of <math>I</math>, and the integral ranges over <math>\mathbb{R}^3</math>.

In the discrete version,<math display="inline">S_w[p] = \sum_r w[r] S_0[p-r]</math>, where

:<math display="block">

S_0[p] =

\begin{bmatrix}

(I_x[p])^2 & I_x[p]I_y[p] & I_x[p]I_z[p] \\[10pt]

I_x[p]I_y[p] & (I_y[p])^2 & I_y[p]I_z[p]\\[10pt]

I_x[p]I_z[p] & I_y[p]I_z[p] & (I_z[p])^2

\end{bmatrix}

</math>

and the sum ranges over a finite set of 3D indices, usually <math>\{-m.. \ldots +m\} \times \{-m.. \ldots +m\} \times \{-m.. \ldots +m\}</math> for some ''{{mvar|m''}}.

===Interpretation===

As in the threetwo-dimensional case, the eigenvalues <math>\lambda_1,\lambda_2,\lambda_3</math> of <math>S_w[p]</math>, and the corresponding eigenvectors <math>e_1\hat{e}_1,e_2\hat{e}_2,e_3\hat{e}_3</math>, summarize the distribution of gradient directions within the neighborhood of ''p'' defined by the window <math>w</math>. This information can be visualized as an [[ellipsoid]] whose semi-axes are equal to the eigenvalues and directed along their corresponding eigenvectors.<ref name="Medioni"/><ref>{{Cite journal | last1=Westin|first1=C.-F. | last2=Maier|first2=S.E. | last3=Mamata|first3=H. | last4=Nabavi|first4=A. | last5=Jolesz|first5=F.A. | last6=Kikinis|first6=R. | date=June 2002 | title=Processing and visualization for diffusion tensor MRI | url=https://linkinghub.elsevier.com/retrieve/pii/S1361841502000531 | journal = Medical Image Analysis | language=en | volume=6 | issue=2 | pages=93–108 | doi=10.1016/S1361-8415(02)00053-1 | pmid=12044998| url-access=subscription }}</ref>

[[File:STgeneric.png|thumb|center|240px|Ellipsoidal representation of the 3D structure tensor.]]

In particular, if the ellipsoid is stretched along one axis only, like a cigar (that is, if <math>\lambda_1</math> is much larger than both <math>\lambda_2</math> and <math>\lambda_3</math>), it means that the gradient in the window is predominantly aligned with the direction <math>e_1</math>, so that the [[isosurface]]s of <math>I</math> tend to be flat and perpendicular to that vector. This situation occurs, for instance, when ''p'' lies on a thin plate-like feature, or on the smooth boundary between two regions with contrasting values.

{| cellborder=0px border=0px style="margin: 1em auto;"

<center>

{| cellborder=0px border=0px

|- valign=top

| [[File:STsurfel.png|thumb|180px|The structure tensor ellipsoid of a surface-like neighborhood ("[[surfel]]"), where <math>\lambda_1 >\!> \lambda_2 \approx \lambda_3</math>.]]||[[File:StepPlane3D.png|thumb|180px|A 3D window straddling a smooth boundary surface between two uniform regions of a 3D image.]]||[[File:StepPlane3DST.png|thumb|180px|The corresponding structure tensor ellipsoid.]]

|}

</center>

If the ellipsoid is flattened in one direction only, like a pancake (that is, if <math>\lambda_3</math> is much smaller than both <math>\lambda_1</math> and <math>\lambda_2</math>), it means that the gradient directions are spread out but perpendicular to <math>e_3</math>; so that the isosurfaces tend to be like tubes parallel to that vector. This situation occurs, for instance, when ''p'' lies on a thin line-like feature, or on a sharp corner of the boundary between two regions with contrasting values.

{| cellborder=0px border=0px style="margin: 1em auto;"

<center>

{| cellborder=0px border=0px

|- valign=top

| [[File:STcurvel.png|thumb|180px|The structure tensor of a line-like neighborhood ("curvel"), where <math>\lambda_1 \approx \lambda_2 >\!> \lambda_3</math>.]]||[[File:curve3D.png|thumb|180px|A 3D window straddling a line-like feature of a 3D image.]]||[[File:curve3DST.png|thumb|180px|The corresponding structure tensor ellipsoid.]]

|}

</center>

Finally, if the ellipsoid is roughly spherical (that is, if <math>\lambda_1\approx\lambda_2\approx\lambda_3</math>), it means that the gradient directions in the window are more or less evenly distributed, with no marked preference; so that the function <math>I</math> is mostly isotropic in that neighborhood. This happens, for instance, when the function has [[spherical symmetry]] in the neighborhood of ''p''. In particular, if the ellipsoid degenerates to a point (that is, if the three eigenvalues are zero), it means that <math>I</math> is constant (has zero gradient) within the window.

{| cellborder=0px border=0px style="margin: 1em auto;"

<center>

{| cellborder=0px border=0px

|- valign=top

| [[File:STball.png|thumb|180px|The structure tensor in an isotropic neighborhood, where <math>\lambda_1\approx\lambda_2\approx\lambda_3</math>.]]||[[File:Sphere3D.png|thumb|180px|A 3D window containing a spherical feature of a 3D image.]]||[[File:Sphere3DST.png|thumb|180px|The corresponding structure tensor ellipsoid.]]

|}

</center>

==The multi-scale structure tensor==

The structure tensor is an important tool in [[scale space]] analysis. The '''multi-scale structure tensor''' (or '''multi-scale second moment matrix''') of a function <math>I</math> is in contrast to other one-parameter scale-space features an image descriptor that is defined over ''two'' scale parameters.

One scale parameter, referred to as ''local scale'' <math>t</math>, is needed for determining the amount of pre-smoothing when computing the image gradient <math>(\nabla I)(x; t)</math>. Another scale parameter, referred to as ''integration scale'' <math>s</math>, is needed for specifying the spatial extent of the window function <math>w(\xi; s)</math> that determines the weights for the region in space over which the components of the outer product of the gradient by itself <math>(\nabla I)(\nabla I)^\text{T}</math> are accumulated.

More precisely, suppose that <math>I</math> is a real-valued signal defined over <math>\mathbb{R}^k</math>. For any local scale <math>t > 0</math>, let a multi-scale representation <math>I(x; t)</math> of this signal be given by <math>I(x; t) = h(x; t)*I(x)</math> where <math>h(x; t)</math> represents a pre-smoothing kernel. Furthermore, let <math>(\nabla I)(x; t)</math> denote the gradient of the [[scale space representation]].

Then, the ''multi-scale structure tensor/second-moment matrix'' is defined by<ref name=lin94book/><ref name=lingar97>{{cite journal

| author1=T. Lindeberg | author2=J. Garding

<ref name=lin94book/><ref name=lingar97>{{cite journal

| name-list-style=amp

|author1=T. Lindeberg |author2=J. Garding

|name-list-style=amp | title=Shape-adapted smoothing in estimation of 3-D depth cues from affine distortions of local 2-D structure

| journal=Image and Vision Computing

| year=1997

| volume=15

| pages=415–434

| url=http://kth.diva-portal.org/smash/record.jsf?pid=diva2%3A472972&dswid=7790

| url=http://www.nada.kth.se/~tony/abstracts/LG94-ECCV.html

| doi=10.1016/S0262-8856(97)01144-X

| issue=6

}}</ref><ref name=garlin96>

J. Garding and T. Lindeberg (1996). ''[http://wwwkth.nadadiva-portal.kth.se/cvaporg/abstractssmash/cvap117record.htmljsf?pid=diva2%3A473351&dswid=-5409 "Direct computation of shape cues using scale-adapted spatial derivative operators]'', International Journal of Computer Vision, volume 17, issue 2, pages 163–191.

</ref>

:<math display="block">

\mu(x; t, s) =

\int_{\xi \in \mathbb{R}^k}

(\nabla I)(x-\xi; t) \, (\nabla I)^\text{T}(x-\xi; t) \,

w(\xi; s) \, d\xi

</math>

Conceptually, one may ask if it would be sufficient to use any self-similar families of smoothing functions <math>h(x; t)</math> and <math>w(\xi; s)</math>. If one naively would apply, for example, a box filter, however, then non-desirable artifacts could easily occur. If one wants the multi-scale structure tensor to be well-behaved over both increasing local scales <math>t</math> and increasing integration scales <math>s</math>, then it can be shown that both the smoothing function and the window function ''have to'' be Gaussian.<ref name=lin94book/> The conditions that specify this uniqueness are similar to the [[scale-space axioms]] that are used for deriving the uniqueness of the Gaussian kernel for a regular Gaussian [[scale space]] of image intensities.

Line 232 ⟶ 227:

A conceptually similar construction can be performed for discrete signals, with the convolution integral replaced by a convolution sum and with the continuous Gaussian kernel <math> g(x; t)</math> replaced by the [[discrete Gaussian kernel]] <math>T(n; t)</math>:

:<math display="block">

\mu(x; t, s) =

\sum_{n \in \mathbb{Z}^k}

(\nabla I)(x-n; t) \, (\nabla I)^\text{T}(x-n; t) \,

w(n; s)

</math>

When quantizing the scale parameters <math>t</math> and <math>s</math> in an actual implementation, a finite geometric progression <math>\alpha^i</math> is usually used, with ''i'' ranging from 0 to some maximum scale index ''m''. Thus, the discrete scale levels will bear certain similarities to [[pyramid (image processing)|image pyramid]], although spatial subsampling may not necessarily be used in order to preserve more accurate data for subsequent processing stages.

==Applications==

The eigenvalues of the structure tensor play a significant role in many image processing algorithms, for problems like [[corner detection]], [[interest point detection]], and [[feature tracking]].<ref name="Medioni">

{{cite conference|author1=M. Nicolescu |author2=G. Medioni |name-list-style=amp |titlechapter=Motion Segmentation with Accurate Boundaries – A Tensor Voting Approach|booktitletitle=Proc. IEEE Computer Vision and Pattern Recognition| volume=1| pages=382–389| year=2003}}

</ref><ref>

{{cite journal

|author=W. Förstner|title=A Feature Based Correspondence Algorithm for Image Processing

|booktitlejournal=International Archives of Photogrammetry and Remote Sensing|volume=26|pages=150–166|year=1986}}

</ref><ref>

{{cite conference|author1=C. Harris |author2=M. Stephens |name-list-style=amp |titlechapter=A Combined Corner and Edge Detector

|booktitletitle=Proc. of the 4th ALVEY Vision Conference|pages=147–151|year=1988}}

</ref><ref>

{{cite journal|author=K. Rohr|title=On 3D Differential Operators for Detecting Point Landmarks

|booktitlejournal=Image and Vision Computing|volume=15|issue=3|pages=219–233|year=1997 | doi=10.1016/S0262-8856(96)01127-4}}

</ref><ref>

{{cite conference|author1=I. Laptev |author2=T. Lindeberg |name-list-style=amp |titlechapter=Space–time interest points

|booktitletitle=International Conference on Computer Vision ICCV'03|chapter-url=ftphttp://ftp.nada.kth.sediva-portal.org/CVAPsmash/reports/LapLin03-ICCVrecord.pdfjsf?pid=diva2%3A442088&dswid=2038 |doi=10.1109/ICCV.2003.1238378|pages=432–439|volume=I|year=2003}}

</ref><ref>

{{cite conference|author=B. Triggs|titlechapter=Detecting Keypoints with Stable Position, Orientation, and Scale under Illumination Changes

|booktitletitle=Proc. European Conference on Computer Vision|volume=4|pages=100–113|year=2004}}

</ref><ref>

{{cite conference|author1=C. Kenney, M. Zuliani |author2=B. Manjunath |name-list-style=amp |titlechapter=An Axiomatic Approach to Corner Detection|booktitletitle=Proc. IEEE Computer Vision and Pattern Recognition|pages=191–197|year=2005}}

</ref> The structure tensor also plays a central role in the [[Lucas–Kanade Optical Flow Method|Lucas-Kanade optical flow algorithm]], and in its extensions to estimate [[affine shape adaptation]];<ref name=lingar97/> where the magnitude of <math>\lambda_2</math> is an indicator of the reliability of the computed result. The tensor has been used for [[scale space]] analysis,<ref name=lin94book/> estimation of local surface orientation from monocular or binocular cues,<ref name=garlin96/> non-linear [[fingerprint enhancement]],<ref>

A. Almansa and T. Lindeberg (2000), ''[http://wwwkth.nadadiva-portal.kth.seorg/cvapsmash/abstracts/cvap226record.htmljsf?pid=diva2%3A338874&dswid=-9161 Enhancement of fingerprint images using shape-adaptated scale-space operators]''. IEEE Transactions on Image Processing, volume 9, number 12, pages 2027–2042.

</ref> [[diffusion-based image processing]],<ref>[http://www.mia.uni-saarland.de/weickert/book.html J. Weickert (1998), Anisotropic diffusion in image processing, Teuber Verlag, Stuttgart.]</ref><ref>

{{cite journal|authorauthor1=D. Tschumperle and| author2= R. Deriche| name-list-style=amp | title=Diffusion PDE'sPDEs on Vector-Valued Images|booktitlejournal=IEEE Signal Processing Magazine|pages=16–25|volume=19|issue=5|doi=10.1109/MSP.2002.1028349 | date=September 2002| bibcode= 2002ISPM...19...16T}}

</ref><ref>

{{cite conference|author1=S. Arseneau |author2=J. Cooperstock |name-list-style=amp |titlechapter=An Asymmetrical Diffusion Framework for Junction Analysis|booktitletitle=British Machine Vision Conference|volume=2|pages=689–698|date=September 2006}}

</ref><ref>

{{cite conference|author1=S. Arseneau |author2=J. Cooperstock |name-list-style=amp |titlechapter=An Improved Representation of Junctions through Asymmetric Tensor Diffusion|booktitletitle=International Symposium on Visual Computing|date=November 2006}}

</ref> and several other image processing problems. The structure tensor can be also applied in [[geology]] to filter [[Seismology|seismic]] data.<ref>{{Cite journal| last1=Yang|first1=Shuai| last2=Chen|first2=Anqing| last3=Chen|first3=Hongde| date=2017-05-25|title=Seismic data filtering using non-local means algorithm based on structure tensor| journal=Open Geosciences| volume=9|issue=1|pages=151–160|doi=10.1515/geo-2017-0013| issn=2391-5447| bibcode=2017OGeo....9...13Y| s2cid=134392619|doi-access=free}}</ref>

===Processing spatio-temporal video data with the structure tensor===

The three-dimensional structure tensor has been used to analyze three-dimensional video data (viewed as a function of ''x'', ''y'', and time ''t'').<ref name="Jahne1993" />

If one in this context aims at image descriptors that are ''invariant'' under [[Galilean transformationstransformation]]s, to make it possible to compare image measurements that have been obtained under variations of a priori unknown image velocities <math>v = (v_x, v_y)^\text{T}</math>

:<math display="block"> \begin{bmatrix} x' \\ y' \\ t' \end{bmatrix} = G \begin{bmatrix} x \\ y \\ t \end{bmatrix} = \begin{bmatrix} x - v_x \, t \\ y - v_y \, t \\ t \end{bmatrix} ,</math>,

it is, however, from a computational viewpoint preferable to parameterize the components in the structure tensor/second-moment matrix <math>S</math> using the notion of ''Galilean diagonalization''<ref name=lin04icpr>

{{cite conference|author1=T. Lindeberg |author2=A. Akbarzadeh |author3=I. Laptev |name-list-style=amp |titlechapter=Galilean-corrected spatio-temporal interest operators|booktitletitle=International Conference on Pattern Recognition ICPR'04|chapter-url=ftphttp://ftp.nada.kth.sediva-portal.org/CVAPsmash/reports/LinAkhLap04-ICPRrecord.pdfjsf?pid=diva2%3A441205&dswid=-545 |doi=10.1109/ICPR.2004.1334004|date=August 2004|volume=I| pages=57–62}}

</ref>

:<math display="block"> S' = R_\text{space}^{-\text{T}} \, G^{-\text{T}} \, S \, G^{-1} \, R_\text{space}^{-1} = \begin{bmatrix} \nu_1 & \, & \, \\ \, & \nu_2 & \, \\ \, & \, & \nu_3 \end{bmatrix} </math>

where <math>G</math> denotes a Galilean transformation of spacetime and <math>R_\text{space}</math> a two-dimensional rotation over the spatial ___domain,

compared to the abovementioned use of eigenvalues of a 3-D structure tensor, which corresponds to an eigenvalue decomposition and a (non-physical) three-dimensional rotation of spacetime

:<math display="block"> S'' = R_\text{spacetime}^{-\text{T}} \, S \, R_\text{spacetime}^{-1} = \begin{bmatrix} \lambda_1 & & \\ & \lambda_2 & \\ & & \lambda_3 \end{bmatrix} .</math>.

To obtain true Galilean invariance, however, also the shape of the spatio-temporal window function needs to be adapted,<ref name=lin04icpr/><ref>

{{cite conference|author1=I. Laptev |author2=T. Lindeberg |name-list-style=amp |title=Velocity adaptation of space–time interest points |booktitleconference=International Conference on Pattern Recognition ICPR'04 |url=http://www.csc.kth.sediva-portal.org/cvapsmash/abstracts/LapLin04-ICPRrecord.htmljsf?pid=diva2%3A451230&dswid=-7763|doi=10.1109/ICPR.2004.971|date=August 2004|volume=I| pages=52–56}}

</ref> corresponding to the transfer of [[affine shape adaptation]]<ref name=lingar97/> from spatial to spatio-temporal image data.

In combination with local spatio-temporal histogram descriptors,<ref>

{{cite conference|author1=I. Laptev |author2=T. Lindeberg |name-list-style=amp |title=Local descriptors for spatio-temporal recognition|booktitleconference=ECCV'04 Workshop on Spatial Coherence for Visual Motion Analysis (Prague, Czech Republic) Springer Lecture Notes in Computer Science|url=http://www.csc.kth.sediva-portal.org/cvapsmash/abstracts/LapLin04-ECCVWSrecord.htmljsf?pid=diva2%3A445261&dswid=-1233| doi=10.1007/11676959|date=May 2004|volume=3667| pages=91–103|url-access=subscription}}

</ref>

these concepts together allow for Galilean invariant recognition of spatio-temporal events.<ref>

{{cite conference|author1=I. Laptev |author2=B. Caputo |author3=C. Schuldt |author4=T. Lindeberg |name-list-style=amp |titlechapter=Local velocity-adapted motion events for spatio-temporal recognition |booktitletitle=Computer Vision and Image Understanding|chapter-url=http://www.csc.kth.sediva-portal.org/cvapsmash/abstracts/LapCapSchLin07-CVIUrecord.htmljsf?pid=diva2%3A335153&dswid=7950 | doi=10.1016/j.cviu.2006.11.023|year=2007|volume=108| pages= 207–229}}</ref>

==See also==