Content deleted Content added
Corrected table entry, it was wrong and inconsistent with (correct) value in paragraph describing it |
ShelfSkewed (talk | contribs) →String searching with don't cares: dab link |
||
| (156 intermediate revisions by more than 100 users not shown) | |||
Line 1:
{{short description|Searching for patterns in text}}
A '''string-searching algorithm''', sometimes called '''string-matching algorithm''', is an [[algorithm]] that searches a body of [[string (computer science)|text]] for portions that match by pattern.
In practice,
==
The most basic case of string searching involves one (often very long) string, sometimes called the ''haystack'', and one (often very short) string, sometimes called the ''needle''. The goal is to find one or more occurrences of the needle within the haystack. For example, one might search for ''to'' within:
Some books are to be tasted, others to be swallowed, and some few to be chewed and digested.
One might request the first occurrence of "to", which is the fourth word; or all occurrences, of which there are 3; or the last, which is the fifth word from the end.
Very commonly, however, various constraints are added. For example, one might want to match the "needle" only where it consists of one (or more) complete words—perhaps defined as ''not'' having other letters immediately adjacent on either side. In that case a search for "hew" or "low" should fail for the example sentence above, even though those literal strings do occur.
Another common example involves "normalization". For many purposes, a search for a phrase such as "to be" should succeed even in places where there is something else intervening between the "to" and the "be":
*More than one space
*Other "whitespace" characters such as tabs, non-breaking spaces, line-breaks, etc.
*Less commonly, a hyphen or soft hyphen
*In structured texts, [[Markup language|tags]] or even arbitrarily large but "parenthetical" things such as footnotes, list-numbers or other markers, embedded images, and so on.
Many symbol systems include characters that are synonymous (at least for some purposes):
*Latin-based alphabets distinguish lower-case from upper-case, but for many purposes string search is expected to ignore the distinction.
*Many languages include [[Typographic ligature|ligature]]s, where one composite character is equivalent to two or more other characters.
*Many writing systems involve [[diacritical marks]] such as accents or [[Vowel pointing (disambiguation)|vowel points]], which may vary in their usage, or be of varying importance in matching.
*DNA sequences can involve [[non-coding]] segments which may be ignored for some purposes, or polymorphisms that lead to no change in the encoded proteins, which may not count as a true difference for some other purposes.
* Some languages have rules where a different character or form of character must be used at the start, middle, or end of words.
Finally, for strings that represent natural language, aspects of the language itself become involved. For example, one might wish to find all occurrences of a "word" despite it having alternate spellings, prefixes or suffixes, etc.
Another more complex type of search is [[regular expression]] searching, where the user constructs a pattern of characters or other symbols, and any match to the pattern should fulfill the search. For example, to catch both the American English word "color" and the British equivalent "colour", instead of searching for two different literal strings, one might use a regular expression such as:
colou?r
where the "?" conventionally makes the preceding character ("u") optional.
This article mainly discusses algorithms for the simpler kinds of string searching.
A similar problem introduced in the field of bioinformatics and genomics is the maximal exact matching (MEM).<ref>{{Cite journal|last1=Kurtz|first1=Stefan|last2=Phillippy|first2=Adam|last3=Delcher|first3=Arthur L|last4=Smoot|first4=Michael|last5=Shumway|first5=Martin|last6=Antonescu|first6=Corina|last7=Salzberg|first7=Steven L|date=2004|title=Versatile and open software for comparing large genomes |journal=Genome Biology |volume=5|issue=2|pages=R12|doi=10.1186/gb-2004-5-2-r12|issn=1465-6906 |pmc= 395750|pmid=14759262 |doi-access=free }}</ref> Given two strings, MEMs are common substrings that cannot be extended left or right without causing a mismatch.<ref>{{Cite journal|last1=Khan|first1=Zia|last2=Bloom|first2=Joshua S.|last3=Kruglyak|first3=Leonid|last4=Singh|first4=Mona|date=2009-07-01|title=A practical algorithm for finding maximal exact matches in large sequence datasets using sparse suffix arrays|url= |journal=Bioinformatics |volume=25|issue=13|pages=1609–1616|doi=10.1093/bioinformatics/btp275 |pmc=2732316|pmid=19389736}}</ref>
== Examples of search algorithms ==
=== Naive string search ===
A simple and inefficient way to see where one string occurs inside another is to check at each index, one by one. First, we see if there is a copy of the needle starting at the first character of the haystack; if not, we look to see if there's a copy of the needle starting at the second character of the haystack, and so forth. In the normal case, we only have to look at one or two characters for each wrong position to see that it is a wrong position, so in the average case, this takes [[Big O notation|O]](''n'' + ''m'') steps, where ''n'' is the length of the haystack and ''m'' is the length of the needle; but in the worst case, searching for a string like "aaaab" in a string like "aaaaaaaaab", it takes [[Big O notation|O]](''nm'')
=== Finite-state-automaton-based search ===
[[Image:DFA search mommy.svg|200px|right]]
In this approach, backtracking is avoided by constructing a [[deterministic finite automaton]] (DFA) that recognizes a stored search string. These are expensive to construct—they are usually created using the [[powerset construction]]—but are very quick to use. For example, the [[deterministic finite automaton|DFA]] shown to the right recognizes the word "MOMMY". This approach is frequently generalized in practice to search for arbitrary [[regular expression]]s.
===Stubs===
[[Knuth–Morris–Pratt algorithm|Knuth–Morris–Pratt]] computes a [[deterministic finite automaton|DFA]] that recognizes inputs with the string to search for as a suffix, [[Boyer–Moore string-search algorithm|Boyer–Moore]] starts searching from the end of the needle, so it can usually jump ahead a whole needle-length at each step. Baeza–Yates keeps track of whether the previous ''j'' characters were a prefix of the search string, and is therefore adaptable to [[fuzzy string searching]]. The [[bitap algorithm]] is an application of Baeza–Yates' approach.
=== Index methods ===
Faster search algorithms preprocess the text. After building a [[substring index]], for example a [[suffix tree]] or [[suffix array]], the occurrences of a pattern can be found quickly. As an example, a suffix tree can be built in <math>\Theta(n)</math> time, and all <math>z</math> occurrences of a pattern can be found in <math>O(m)</math> time under the assumption that the alphabet has a constant size and all inner nodes in the suffix tree know what leaves are underneath them. The latter can be accomplished by running a [[Depth-first search|DFS algorithm]] from the root of the suffix tree.
=== Other variants ===
Some search methods, for instance [[trigram search]], are intended to find a "closeness" score between the search string and the text rather than a "match/non-match". These are sometimes called [[Approximate string matching|"fuzzy" searches]].
== Classification of search algorithms ==
=== Classification by a number of patterns ===
The various [[algorithm]]s can be classified by the number of patterns each uses.
==== Single-pattern algorithms ====
In the following compilation, ''m'' is the length of the pattern, ''n'' the length of the searchable text, and ''k'' = |Σ| is the size of the alphabet.
{| class="wikitable"
|-
Line 16 ⟶ 68:
! Preprocessing time
! Matching time{{ref|Asymptotic times}}
! Space
|-
! Naïve
| none
| Θ(n+m) in average,<br/> O(mn)
| none
|-
! Automaton-based matching
| Θ(
| Θ(n)
| Θ(km)
|-
! [[
| Θ(m)
| Θ(n) in average,<br/> O(mn) at worst
| O(1)
|-
! [[Knuth–Morris–Pratt algorithm|Knuth–Morris–Pratt]]
| Θ(m)
| Θ(n)
| Θ(m)
|-
! [[Boyer–Moore string
| Θ(m + k)
|
| Θ(k)
|-
! [[Two-way string-matching algorithm|Two-way algorithm]]<ref>{{cite journal |last1=Crochemore |first1=Maxime |last2=Perrin |first2=Dominique |title=Two-way string-matching |journal=Journal of the ACM |date=1 July 1991 |volume=38 |issue=3 |pages=650–674 |doi=10.1145/116825.116845 |s2cid=15055316 |url=http://monge.univ-mlv.fr/~mac/Articles-PDF/CP-1991-jacm.pdf |access-date=5 April 2019 |archive-date=24 November 2021 |archive-url=https://web.archive.org/web/20211124025145/http://monge.univ-mlv.fr/~mac/Articles-PDF/CP-1991-jacm.pdf |url-status=live }}</ref>{{ref|libc}}
| Θ(m
| O(n)
| O(log(m))
|-
! Backward Non-Deterministic [[Suffix automaton|DAWG]] Matching (BNDM)<ref>{{cite book |last1=Navarro |first1=Gonzalo |last2=Raffinot |first2=Mathieu |title=Combinatorial Pattern Matching |chapter=A bit-parallel approach to suffix automata: Fast extended string matching |date=1998 |volume=1448 |pages=14–33 |doi=10.1007/bfb0030778 |chapter-url=https://users.dcc.uchile.cl/~gnavarro/ps/cpm98.pdf |publisher=Springer Berlin Heidelberg |series=Lecture Notes in Computer Science |isbn=978-3-540-64739-3 |access-date=2019-11-22 |archive-date=2019-01-05 |archive-url=https://web.archive.org/web/20190105101910/https://users.dcc.uchile.cl/~gnavarro/ps/cpm98.pdf |url-status=live }}</ref>{{ref|fuzzy+regexp}}
| O(m)
| Ω(n/m) at best,<br/> O(mn) at worst
|
|-
! Backward Oracle Matching (BOM)<ref>{{cite book |last1=Fan |first1=H. |last2=Yao |first2=N. |last3=Ma |first3=H. |title=2009 Fourth International Conference on Internet Computing for Science and Engineering |chapter=Fast Variants of the Backward-Oracle-Marching Algorithm |date=December 2009 |pages=56–59 |doi=10.1109/ICICSE.2009.53 |chapter-url=https://pdfs.semanticscholar.org/8d81/94c293f8a81394ba545d09bd6ec711ad4c17.pdf |isbn=978-1-4244-6754-9 |s2cid=6073627 |access-date=2019-11-22 |archive-date=2022-05-10 |archive-url=https://web.archive.org/web/20220510030242/https://www.semanticscholar.org/paper/Efficient-Variants-of-the-Backward-Oracle-Matching-Faro-Lecroq/d24e4bad432e5b88b4fb752ba868e3d3e609a2c4?p2df |url-status=live }}</ref>
| O(m)
| O(mn)
|
|}
:1.{{note|Asymptotic times}}Asymptotic times are expressed using [[Big O notation|O, Ω, and Θ notation]].
:2.{{note|libc}}Used to implement the ''memmem'' and [[C string handling#Functions|''strstr'']] search functions in the [[glibc]]<ref>{{cite web |title=glibc/string/str-two-way.h |url=https://code.woboq.org/userspace/glibc/string/str-two-way.h.html |access-date=2022-03-22 |archive-date=2020-09-20 |archive-url=https://web.archive.org/web/20200920180414/https://code.woboq.org/userspace/glibc/string/str-two-way.h.html |url-status=live }}</ref> and [[musl]]<ref>{{cite web |title=musl/src/string/memmem.c |url=http://git.musl-libc.org/cgit/musl/tree/src/string/memmem.c |access-date=23 November 2019 |archive-date=1 October 2020 |archive-url=https://web.archive.org/web/20201001184748/http://git.musl-libc.org/cgit/musl/tree/src/string/memmem.c |url-status=live }}</ref> [[C standard libraries]].
:3.{{note|fuzzy+regexp}}Can be extended to handle [[approximate string matching]] and (potentially-infinite) sets of patterns represented as [[regular language]]s.{{Citation needed|date=March 2022|reason=No reference found for the 2-way algorithm}}
The '''[[Boyer–Moore string
==== Algorithms using a finite set of patterns ====
In the following compilation, ''M'' is the length of the longest pattern, ''m'' their total length, ''n'' the length of the searchable text, ''o'' the number of occurrences.
{| class="wikitable"
|-
! Algorithm
! Extension of
! Preprocessing time
! Matching time{{ref|Asymptotic times}}
! Space
|-
! [[Aho–Corasick algorithm|Aho–Corasick]]
| [[Knuth–Morris–Pratt algorithm|Knuth–Morris–Pratt]]
| Θ(m)
| Θ(n + o)
| Θ(m)
|-
! [[Commentz-Walter algorithm|Commentz-Walter]]
| [[Boyer–Moore string-search algorithm|Boyer-Moore]]
| Θ(m)
| Θ(M * n) worst case <br/> sublinear in average<ref name="Commentz-Walter"/>
| Θ(m)
|-
! Set-BOM
| Backward Oracle Matching
|
|
|
|}
==== Algorithms using an infinite number of patterns ====
Naturally, the patterns can not be enumerated finitely in this case. They are represented usually by a [[regular grammar]] or [[regular expression]].
=== Classification by the use of preprocessing programs ===
Other classification approaches are possible. One of the most common uses preprocessing as main criteria.
{| class="wikitable"
|+Classes of string searching algorithms<ref>Melichar, Borivoj, Jan Holub, and J. Polcar. Text Searching Algorithms. Volume I: Forward String Matching. Vol. 1. 2 vols., 2005. http://stringology.org/athens/TextSearchingAlgorithms/ {{Webarchive|url=https://web.archive.org/web/20160304074815/http://stringology.org/athens/TextSearchingAlgorithms/ |date=2016-03-04 }}.</ref>
!
!Text not preprocessed
Line 69 ⟶ 164:
! Patterns preprocessed
| Constructed search engines
| Signature methods<ref>{{citation |last1=Litwin |first1=Witold |last2=Mokadem |first2=Riad |last3=Rigaux |first3=Philippe |last4=Schwarz |first4=Thomas |url=https://www.vldb.org/conf/2007/papers/research/p207-litwin.pdf |title=Fast nGram-Based String Search Over Data Encoded Using Algebraic Signatures |year=2007 |publisher=[[International Conference on Very Large Data Bases]]}}</ref>
|}
===
Another one classifies the algorithms by their matching strategy:<ref>{{citation|surname1=Gonzalo Navarro|surname2=Mathieu Raffinot|title=Flexible Pattern Matching Strings: Practical On-Line Search Algorithms for Texts and Biological Sequences |isbn=978-0-521-03993-2|date= 2008|publisher=Cambridge University Press }}</ref>
* Match the prefix first (Knuth–Morris–Pratt, Shift-And, Aho–Corasick)
* Match the suffix first (Boyer–Moore and variants, Commentz-Walter)
* Match the best factor first (BNDM, BOM, Set-BOM)
* Other strategy (Naïve, Rabin–Karp, Vectorized)
===Real-time string matching===
In real-time string matching, one requires the matcher to output a response after reading each character of the text, that indicates
whether this is the last character of a match. The response has to be given within constant time.
The requirement regarding preprocessing vary: O(''m'') preprocessing may be allowed after the pattern is read (but before the reading of
the text), or a stricter requirement may be posed according to which the matcher has to also pause for at most a constant time after reading any character of the pattern (including the last).
For the more lenient version, if one does not mind that the preprocessing time and memory requirement dependend on the size of the alphabet, a real-time solution is provided by
automaton matching.
[[Zvi Galil]] developed a method to turn certain algorithms into real-time algorithms, and applied it to produce a variant of the KMP
matcher that runs in real time under the strict requirement.<ref>{{Cite journal
| last = Galil
| first = Zvi
| title = String matching in real time
| journal = Journal of the ACM
| volume = 28
| issue = 1
| pages = 134–149
| year = 1981
| doi = 10.1145/322234.322244
| pmid =
| url =
}}</ref>
==String searching with don't cares==
In this version of the string searching problem, there is a special symbol, ø (read: don't care), which can match any other symbol (including another ø).
Don't care symbols can appear either in the pattern or in the text. In 2002, an algorithm for this problem that runs in <math>O(n\log m)</math> time has been
given by [[Richard Cole]] and [[Ramesh Hariharan]], improving on a solution from 1973 by [[Michael J. Fischer|Fischer]] and [[Michael S. Paterson|Paterson]]
that has complexity <math>O(n\log m\log k)</math>, where ''k'' is the size of the alphabet.<ref>{{Cite conference
| last1 = Cole
| first1 = Richard
| last2 = Hariharan
| first2 = Ramesh
| title = Verifying candidate matches in sparse and wildcard matching
| book-title = Proceedings of the thiry-fourth annual ACM symposium on Theory of computing
| editor =
| publisher =
| ___location =
| pages = 592–601
| year = 2002
| url =
| doi =
}}</ref> Another algorithm, claimed simpler, has been proposed by [[Peter Clifford (mathematician)|Clifford]] and [[Raphaël Clifford|Clifford]].<ref>{{cite journal |last1=Clifford |first1=Peter |last2=Clifford |first2=Raphaël |title=Simple deterministic wildcard matching |journal=Information Processing Letters |date=January 2007 |volume=101 |issue=2 |pages=53–54 |doi=10.1016/j.ipl.2006.08.002}}</ref>
==See also==
*[[Sequence alignment]]
*[[Graph matching]]
*[[Pattern matching]]
*[[Compressed pattern matching]]
*[[Matching wildcards]]
*[[Approximate string matching]]
*[[Full-text search]]
==References==
<references
<ref name="Commentz-Walter">{{cite conference
| last = Commentz-Walter
| first = Beate
| title = A String Matching Algorithm Fast on the Average
| url = http://www.hs-albsig.de/studium/wirtschaftsinformatik/Documents/commentzwalterextab.pdf
| archive-url = https://web.archive.org/web/20171010223532/http://www.hs-albsig.de/studium/wirtschaftsinformatik/Documents/commentzwalterextab.pdf
| archive-date = 2017-10-10
| conference = [[International Colloquium on Automata, Languages and Programming]]
| ___location = Graz, Austria
| series = [[Lecture Notes in Computer Science|LNCS]]
| volume = 71
| pages = 118–132
| year = 1979
| publisher = Springer
| doi = 10.1007/3-540-09510-1_10
| isbn = 3-540-09510-1 }}</ref>
</references>
*R. S. Boyer and J. S. Moore, ''[http://www.cs.utexas.edu/~moore/publications/fstrpos.pdf A fast string searching algorithm],'' Carom. ACM 20, (10), 262–272(1977).
* [[Thomas H. Cormen]], [[Charles E. Leiserson]], [[Ronald L. Rivest]], and [[Clifford Stein]]. ''[[Introduction to Algorithms]]'',
==External links==
{{Commonscat}}
*[http://www.cs.ucr.edu/%7Estelo/pattern.html Huge list of pattern matching links] Last updated: 12/27/2008 20:18:38
*
*[https://xlinux.nist.gov/dads/HTML/stringMatching.html NIST list of string-matching algorithms]
*[http://johannburkard.de/software/stringsearch/ StringSearch – high-performance pattern matching algorithms in Java] – Implementations of many String-Matching-Algorithms in Java (BNDM, Boyer-Moore-Horspool, Boyer-Moore-Horspool-Raita, Shift-Or)
*[http://stringsandchars.amygdalum.net/ StringsAndChars] – Implementations of many String-Matching-Algorithms (for single and multiple patterns) in Java
*[http://www-igm.univ-mlv.fr/~lecroq/string/index.html Exact String Matching Algorithms] — Animation in Java, Detailed description and C implementation of many algorithms.
*[http://www.cs.ucr.edu/~stelo/cpm/cpm04/35_Navarro.pdf (PDF) Improved Single and Multiple Approximate String Matching] {{Webarchive|url=https://web.archive.org/web/20170311221134/http://www.cs.ucr.edu/~stelo/cpm/cpm04/35_Navarro.pdf |date=2017-03-11 }}
*[https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2647288/ Kalign2: high-performance multiple alignment of protein and nucleotide sequences allowing external features]
*[https://www.codeproject.com/Articles/5282980/Fastest-Fulltext-Vector-Scalar-Exact-Searcher NyoTengu – high-performance pattern matching algorithm in C] – Implementations of Vector and Scalar String-Matching-Algorithms in C
*[https://arxiv.org/html/2502.20338v3 Nathaniel K. Brown, et al.: "KeBaB: k-mer based breaking for finding long MEMs", arXiv:2502.20338v3 (09 Jun 2025).]
{{strings}}
| |||