Hierarchical clustering: Difference between revisions

Revision as of 11:25, 23 May 2025

Latest revision as of 00:47, 31 July 2025

(7 intermediate revisions by 6 users not shown)

Line 3:

{{Machine learning|Clustering}}

In [[data mining]] and [[statistics]], '''hierarchical clustering'''<ref name="HC">{{cite book |first=Frank |last=Nielsen | title=Introduction to HPC with MPI for Data Science | year=2016 | publisher=Springer |isbn=978-3-319-21903-5 |pages=195–211

|chapter=8. Hierarchical Clustering | url=https://www.springer.com/gp/book/9783319219028 |chapter-url=https://www.researchgate.net/publication/314700681 }}</ref> (also called '''hierarchical cluster analysis''' or '''HCA''') is a method of [[cluster analysis]] that seeks to build a [[hierarchy]] of clusters. Strategies for hierarchical clustering generally fall into two categories:

* '''Agglomerative''': Agglomerative: Agglomerative clustering, often referred to as a "bottom-up" approach, begins with each data point as an individual cluster. At each step, the algorithm merges the two most similar clusters based on a chosen distance metric (e.g., Euclidean distance) and linkage criterion (e.g., single-linkage, complete-linkage).<ref name=":4">{{Cite journal |lastlast1=Murtagh |firstfirst1=Fionn |last2=Contreras |first2=Pedro |date=2012 |title=Algorithms for hierarchical clustering: an overview |url=https://wires.onlinelibrary.wiley.com/doi/10.1002/widm.53 |journal=WIREs Data Mining and Knowledge Discovery |language=en |volume=2 |issue=1 |pages=86–97 |doi=10.1002/widm.53 |issn=1942-4795|url-access=subscription }}</ref>. This process continues until all data points are combined into a single cluster or a stopping criterion is met. Agglomerative methods are more commonly used due to their simplicity and computational efficiency for small to medium-sized datasets .<ref>{{Cite journal |last=Mojena |first=R. |date=1977-04-01 |title=Hierarchical grouping methods and stopping rules: an evaluation |url=https://academic.oup.com/comjnl/article-lookup/doi/10.1093/comjnl/20.4.359 |journal=The Computer Journal |language=en |volume=20 |issue=4 |pages=359–363 |doi=10.1093/comjnl/20.4.359 |issn=0010-4620}}</ref>.

* '''Divisive''': Divisive clustering, known as a "top-down" approach, starts with all data points in a single cluster and recursively splits the cluster into smaller ones. At each step, the algorithm selects a cluster and divides it into two or more subsets, often using a criterion such as maximizing the distance between resulting clusters. Divisive methods are less common but can be useful when the goal is to identify large, distinct clusters first.

In general, the merges and splits are determined in a [[greedy algorithm|greedy]] manner. The results of hierarchical clustering<ref>{{cite book |firstname=Frank"HC" |last=Nielsen/> |are title=Introductionusually topresented HPCin witha MPI[[dendrogram]]. for Data Science | year=2016 | publisher=Springer |isbn=978-3-319-21903-5 |pages=195–211

|chapter=8. Hierarchical Clustering | url=https://www.springer.com/gp/book/9783319219028 |chapter-url=https://www.researchgate.net/publication/314700681 }}</ref> are usually presented in a [[dendrogram]].

Hierarchical clustering has the distinct advantage that any valid measure of distance can be used. In fact, the observations themselves are not required: all that is used is a [[distance matrix|matrix of distances]]. On the other hand, except for the special case of single-linkage distance, none of the algorithms (except exhaustive search in <math>\mathcal{O}(2^n)</math>) can be guaranteed to find the optimum solution.{{cn|date=October 2024}}

Line 19 ⟶ 18:

== Cluster Linkage ==

In order to decide which clusters should be combined (for agglomerative), or where a cluster should be split (for divisive), a measure of dissimilarity between sets of observations is required. In most methods of hierarchical clustering, this is achieved by use of an appropriate [[distance]] ''d'', such as the Euclidean distance, between ''single'' observations of the data set, and a linkage criterion, which specifies the dissimilarity of ''sets'' as a function of the pairwise distances of observations in the sets. The choice of metric as well as linkage can have a major impact on the result of the clustering, where the lower level metric determines which objects are most [[similarity measure|similar]], whereas the linkage criterion influences the shape of the clusters <ref name=":5" />. For example, complete-linkage tends to produce more spherical clusters than single-linkage.

The linkage criterion determines the distance between sets of observations as a function of the pairwise distances between observations.

Line 49 ⟶ 48:

| <math>\sqrt[p]{\frac{1}{|A|\cdot|B|} \sum_{a \in A }\sum_{ b \in B} d(a,b)^p}, p\neq 0</math>

|-

|[[Ward's method|Ward linkage]],<ref name="wards method">{{cite journal |last=Ward |first=Joe H. |year=1963 |title=Hierarchical Grouping to Optimize an Objective Function |journal=Journal of the American Statistical Association |volume=58 |issue=301 |pages=236–244 |doi=10.2307/2282967 |jstor=2282967 |mr=0148188}}</ref> Minimum Increase of Sum of Squares (MISSQ)<ref name=":0">{{Citation |last=Podani |first=János |title=New combinatorial clustering methods |date=1989 |url=https://doi.org/10.1007/978-94-009-2432-1_5 |work=Numerical syntaxonomy |pages=61–77 |editor-last=Mucina |editor-first=L. |place=Dordrecht |publisher=Springer Netherlands |language=en |doi=10.1007/978-94-009-2432-1_5 |isbn=978-94-009-2432-1 |access-date=2022-11-04 |editor2-last=Dale |editor2-first=M. B.|url-access=subscription }}</ref>

|<math>\frac{|A|\cdot|B|}{|A\cup B|} \lVert \mu_A - \mu_B \rVert ^2

= \sum_{x\in A\cup B} \lVert x - \mu_{A\cup B} \rVert^2

Line 77 ⟶ 76:

- \min_{m\in B} \sum_{y\in B} d(m,y)</math>

|-

|Medoid linkage<ref>{{Cite conference |last1=Miyamoto |first1=Sadaaki |last2=Kaizu |first2=Yousuke |last3=Endo |first3=Yasunori |date=2016 |title=Hierarchical and Non-Hierarchical Medoid Clustering Using Asymmetric Similarity Measures |url=https://ieeexplore.ieee.org/document/7801678 |conference=2016 Joint 8th International Conference on Soft Computing and Intelligent Systems (SCIS) and 17th International Symposium on Advanced Intelligent Systems (ISIS) |pages=400–403 |doi=10.1109/SCIS-ISIS.2016.0091}}</ref><ref>{{Cite conference |date=2016 |title=Visual Clutter Reduction through Hierarchy-based Projection of High-dimensional Labeled Data| conference=Graphics Interface |url=https://graphicsinterface.org/wp-content/uploads/gi2016-14.pdf | first1=Dominik|last1=Herr|first2=Qi|last2=Han|first3=Steffen|last3=Lohmann| first4=Thomas |last4=Ertl |access-date=2022-11-04 |website=Graphics Interface |language=en-CA |doi=10.20380/gi2016.14}}</ref>

|<math>d(m_A, m_B)</math> where <math>m_A</math>, <math>m_B</math> are the medoids of the previous clusters

|-

Line 87 ⟶ 86:

* The probability that candidate clusters spawn from the same distribution function (V-linkage).

* The product of in-degree and out-degree on a k-nearest-neighbour graph (graph degree linkage).<ref>{{Cite book|last1=Zhang|first1=Wei|last2=Wang|first2=Xiaogang|last3=Zhao|first3=Deli|last4=Tang|first4=Xiaoou|title=Computer Vision – ECCV 2012 |chapter=Graph Degree Linkage: Agglomerative Clustering on a Directed Graph |date=2012|editor-last=Fitzgibbon|editor-first=Andrew|editor2-last=Lazebnik|editor2-first=Svetlana|editor2-link= Svetlana Lazebnik |editor3-last=Perona|editor3-first=Pietro|editor4-last=Sato|editor4-first=Yoichi|editor5-last=Schmid|editor5-first=Cordelia|series=Lecture Notes in Computer Science|language=en|publisher=Springer Berlin Heidelberg|volume=7572|pages=428–441|doi=10.1007/978-3-642-33718-5_31|isbn=9783642337185|arxiv=1208.5092|bibcode=2012arXiv1208.5092Z|s2cid=14751}} See also: https://github.com/waynezhanghk/gacluster</ref>

* The increment of some cluster descriptor (i.e., a quantity defined for measuring the quality of a cluster) after merging two clusters.<ref>{{cite journal |first1=W. |last1=Zhang |first2=D. |last2=Zhao |first3=X. |last3=Wang |title=Agglomerative clustering via maximum incremental path integral |journal=Pattern Recognition |volume=46 |issue=11 |pages=3056–65 |date=2013 |doi=10.1016/j.patcog.2013.04.013 |citeseerx=10.1.1.719.5355 |bibcode=2013PatRe..46.3056Z}}</ref><ref>{{cite book |last1=Zhao |first1=D. |last2=Tang |first2=X. |chapter=Cyclizing clusters via zeta function of a graph |chapter-url= |title=NIPS'08: Proceedings of the 21st International Conference on Neural Information Processing Systems |date=2008 |isbn=9781605609492 |pages=1953–60 |publisher=Curran |citeseerx=10.1.1.945.1649}}</ref><ref>{{cite journal |first1=Y. |last1=Ma |first2=H. |last2=Derksen |first3=W. |last3=Hong |first4=J. |last4=Wright |title=Segmentation of Multivariate Mixed Data via Lossy Data Coding and Compression |journal=IEEE Transactions on Pattern Analysis and Machine Intelligence |volume=29 |issue=9 |pages=1546–62 |date=2007 |doi=10.1109/TPAMI.2007.1085 |pmid=17627043 |bibcode=2007ITPAM..29.1546M |hdl=2142/99597 |s2cid=4591894 |hdl-access=free }}</ref>

== Agglomerative clustering example ==

Line 141 ⟶ 140:

The dendrogram of DIANA can be constructed by letting the splinter group <math>C_\textrm{new}</math> be a child of the hollowed-out cluster <math>C_*</math> each time. This constructs a tree with <math>C_0</math> as its root and <math>n</math> unique single-object clusters as its leaves.

== Greedy Nature of the Algorithm ==

Hierarchical clustering is often described as a greedy algorithm because it makes a series of locally optimal choices without reconsidering previous steps. At each iteration, it merges the two clusters that are closest together based on a selected distance metric, always choosing the best immediate option available. This approach is "greedy" because it seeks to optimize the current decision rather than planning for the best possible overall clustering <ref name=":4" />. Once two clusters are merged, the decision is final and irreversible, without the possibility of backtracking, which can lead to suboptimal results if earlier choices were not ideal. Despite this, the greedy nature of hierarchical clustering makes it computationally efficient and simple to implement, though it may not always capture the true underlying structure of complex datasets <ref name=":5" />.

== Limitations ==

Hierarchical clustering, particularly in its standard agglomerative form, presents several notable limitations: (a) Time Complexity: Hierarchical clustering, especially in its basic agglomerative form, has a high time complexity of  O(n³). This becomes a significant bottleneck for large datasets, limiting its scalability <ref name="CLINK2">{{cite journal |author=D. Defays |year=1977 |title=An efficient algorithm for a complete-link method |journal=The Computer Journal |publisher=British Computer Society |volume=20 |issue=4 |pages=364–6 |doi=10.1093/comjnl/20.4.364 |doi-access=}}</ref>. (b) Scalability: Due to the time and space complexity, hierarchical clustering algorithms struggle to handle very large datasets efficiently <ref name=DE/>  (c) Sensitivity to Noise and Outliers: Hierarchical clustering methods can be sensitive to noise and outliers in the data, which can lead to the formation of inaccurate or misleading cluster hierarchies <ref name="SLINK2">{{cite journal |author=R. Sibson |year=1973 |title=SLINK: an optimally efficient algorithm for the single-link cluster method |url=http://www.cs.gsu.edu/~wkim/index_files/papers/sibson.pdf |journal=The Computer Journal |publisher=British Computer Society |volume=16 |issue=1 |pages=30–34 |doi=10.1093/comjnl/16.1.30 |doi-access=free}}</ref>. (d) Difficulty with High-Dimensional Data: In high-dimensional spaces, hierarchical clustering can face challenges due to the curse of dimensionality, where data points become sparse, and distance measures become less meaningful. This can result in poorly defined clusters<ref name=":6" /><ref name=":3">{{Cite conference |last1=Herr |first1=Dominik |last2=Han |first2=Qi |last3=Lohmann |first3=Steffen |last4=Ertl |first4=Thomas |date=2016 |title=Visual Clutter Reduction through Hierarchy-based Projection of High-dimensional Labeled Data |url=https://graphicsinterface.org/wp-content/uploads/gi2016-14.pdf |conference=Graphics Interface |language=en-CA |doi=10.20380/gi2016.14 |access-date=2022-11-04 |website=Graphics Interface}}</ref>. (e) Inability to Handle Non-Convex Shapes and Varying Densities: Traditional hierarchical clustering methods, like many other clustering algorithms, often assume that clusters are convex and have similar densities. They may struggle to accurately identify clusters with non-convex shapes or varying densities <ref name=":5">{{Cite journal |last=Wani |first=Aasim Ayaz |date=2024-08-29 |title=Comprehensive analysis of clustering algorithms: exploring limitations and innovative solutions |journal=PeerJ Computer Science |language=en |volume=10 |pages=e2286 |doi=10.7717/peerj-cs.2286 |issn=2376-5992 |pmc=11419652 |pmid=39314716 |doi-access=free}}</ref>.

== Software ==