Compound-term processing: Difference between revisions

Revision as of 21:32, 20 June 2008

Latest revision as of 00:05, 1 January 2021

(51 intermediate revisions by 33 users not shown)

Line 1:

'''Compound -term processing,''' is the name that is used for a category of techniques in [[Informationinformation retrieval|information-retrieval]], applicationsis thatsearch performsresult matching on the basis of [[compound termsterm]]s. Compound terms are searchbuilt termsby thatcombining containtwo or more thansimple one word,terms; for example, "triple" is a single word term, but "triple heart bypass" is a compound term.

Compound -term processing is a new approach to an old problem: how tocan one improve the relevance of search results without missing anything important whilstwhile maintaining ease of use.? ByUsing formingthis compound (i.e. multi-word) terms and placing these in the search engine's index the search can be performed withtechnique, a higher degree of accuracy because the ambiguity inherent in single words is no longer a problem. A search for ''survival rates following a triple heart bypass in elderly people'' will locate documents about this topic even if this precise phrase is not contained in any document. AThis can be performed by a [[concept search]], usingwhich "Compounditself Termuses Processing"compound-term canprocessing. This will extract the key concepts automatically (in this case "survival rates" and, "triple heart bypass" and "elderly people") and use these concepts to select the most relevant documents.

In August 2003 [[Concept Searching Limited]] introduced the idea of using Compound term processing via an article published in INFORMATION MANAGEMENT AND TECHNOLOGY (VOL 36 PART 4). A British Library Direct catalogue entry can be found here: <ref>[http://direct.bl.uk/bld/PlaceOrder.do?UIN=138451913&ETOC=RN] British Library Direct catalogue entry</ref>.

== Techniques ==

The complete original article can also be downloaded from here: <ref>[http://www.conceptsearching.com/Web/UserFiles/File/Concept%20Searching%20Lateral%20Thinking.pdf] Lateral Thinking in Information Retrieval</ref>.

In August 2003, [[Concept Searching Limited]] introduced the idea of using statistical compound-term processing.<ref>{{cite journal|url=http://www.conceptsearching.com/Web/UserFiles/File/Concept%20Searching%20Lateral%20Thinking.pdf|title=Lateral Thinking in Information Retrieval|journal=Information Management and Technology|volume=36 PART 4|access-date=2008-06-20|archive-url=https://web.archive.org/web/20171115145846/https://www.conceptsearching.com/Web/UserFiles/File/Concept%20Searching%20Lateral%20Thinking.pdf|archive-date=2017-11-15|url-status=dead}} The British Library Direct catalogue entry can be found here:[http://direct.bl.uk/bld/PlaceOrder.do?UIN=138451913&ETOC=RN] {{Webarchive|url=https://web.archive.org/web/20120210133832/http://direct.bl.uk/bld/PlaceOrder.do?UIN=138451913&ETOC=RN |date=2012-02-10 }}</ref>

Compound term processing is important because it allows search (and other Information Retrieval) applications to perform their matching on the basis of multi-word concepts rather than single words in isolation which can be highly ambiguous.

CLAMOUR is a European collaborative project which aims to find a better way to classify when collecting and disseminating industrial information and statistics. CLAMOUR appears to use a linguistic approach, rather than one based on [[statistical model|statistical modelling]].<ref>[http://webarchive.nationalarchives.gov.uk/20040117000117/statistics.gov.uk/methods_quality/clamour/default.asp] National Statistics CLAMOUR project</ref>

Most search engines simply look for documents that contain the words that the user enters into the search box (aka "keyword search" engines). [[Boolean search]] engines add a degree of sophistication by allowing the user to specify additional requirements but most users struggle to comprehend and use the necessary syntax (e.g. Tiger NEAR Woods AND (golf OR golfing) NOT Volkswagen). [[Phrase search]] is easier to understand but can lead to many useful documents being missed if they do not contain the exact phrase specified.

== History ==

Techniques for probabilistic weighting of single word terms dates back to at least 1976 and the landmark publication by Stephen Robertson and [[Karen Spärck Jones]]: Relevance weighting of search terms originally published in the Journal of the American Society for Information Science. <ref> [http://www.soi.city.ac.uk/~ser/papers/RSJ76.pdf] Relevance weighting of search terms</ref> Professor Stephen Robertson has stated repeatedly that the assumption of word independence is “patently not justified” and exists simply as a “matter of mathematical convenience”.

Techniques for probabilistic weighting of single word terms date back to at least 1976 in the landmark publication by [[Stephen Robertson (computer scientist)|Stephen E. Robertson]] and [[Karen Spärck Jones]].<ref>{{Cite journal | doi = 10.1002/asi.4630270302| title = Relevance weighting of search terms| journal = Journal of the American Society for Information Science| volume = 27| issue = 3| pages = 129| year = 1976| last1 = Robertson | first1 = S. E. | authorlink1 = Stephen Robertson (computer scientist)| last2 = Spärck Jones | first2 = K. | authorlink2 = Karen Spärck Jones}}</ref> Robertson stated that the assumption of word independence is not justified and exists as a matter of mathematical convenience. His objection to the term independence is not a new idea, dating back to at least 1964 when H. H. Williams stated that "[t]he assumption of independence of words in a document is usually made as a matter of mathematical convenience".<ref>{{cite journal |last=WILLIAMS |first=J.H. |title=Results of classifying documents with multiple discriminant functions |url=http://oai.dtic.mil/oai/oai?verb=getRecord&metadataPrefix=html&identifier=AD0612272 |journal=Statistical Association Methods for Mechanized Documentation, National Bureau of Standards |___location=Washington |pages=217–224 |year=1965 |access-date=2015-05-21 |archive-url=https://web.archive.org/web/20110717145048/http://oai.dtic.mil/oai/oai?verb=getRecord |archive-date=2011-07-17 |url-status=dead }}</ref>

Compound term processing is a new approach to an old problem: how to improve the relevance of search results without missing anything important whilst maintaining ease of use. By forming compound (i.e. multi-word) terms and placing these in the search engine's index the search can be performed with a higher degree of accuracy because the ambiguity inherent in single words is no longer a problem. A search for ''survival rates following a triple heart bypass'' will locate documents about this topic even if this precise phrase is not contained in any document. A [[concept search]] using "Compound Term Processing" can extract the key concepts automatically (in this case "survival rates" and "triple heart bypass") and use these to select the most relevant documents.

In 2004, Anna Lynn Patterson filed a number of patents on the"phrase-based subjectsearching ofin "Phrasean based indexing andinformation retrieval system" and<ref>{{patent|US|20060031195}}</ref> to which [[Google]] subsequently acquired the rights. A full discussion of the patents can be found here: <ref>[http://www.webmasterwomanseobythesea.com/search-engines2012/phrase02/google-basedacquires-indexing.htmlcuil-patent-applications/ WebmasterGoogle Woman].Acquires Cuil ThePatent patents themselves can be found online, for example: <ref>[http://appft1.uspto.gov/netacgi/nph-Parser?Sect1=PTO1&Sect2=HITOFF&d=PG01&p=1&u=%2Fnetahtml%2FPTO%2Fsrchnum.html&r=1&f=G&l=50&s1=%2220060031195%22.PGNR.&OS=DN/20060031195&RS=DN/20060031195Applications] US Patent: 20060031195</ref>.

== Adaptability ==

Compound term processing is more adaptive than the "phrase based indexing and retrieval" detailed by Anna Lynn Patterson in her patent applications. The "phrase based indexing" is targeted at searching the World Wide Web where Google can utilise their extensive statistical knowledge of common searches to identify candidate phrases. Compound term processing is more suited to [[Enterprise Search]] applications where such a priori knowledge is not available.

CompoundStatistical compound-term processing is more adaptiveadaptable than the "phraseprocess based indexing and retrieval" detaileddescribed by Anna Lynn Patterson in her patent applications. The "phrase basedHer indexing"process is targeted at searching the [[World Wide Web]] where Google can utilise theiran extensive statistical knowledge of common searches can be used to identify candidate phrases. CompoundStatistical compound term processing is more suited to [[Enterpriseenterprise Searchsearch]] applications where such [[A priori and a posteriori|a priori]] knowledge is not available.

Statistical compound-term processing is also more adaptable than the linguistic approach taken by the CLAMOUR project, which must consider the syntactic properties of the terms (i.e. part of speech, gender, number, etc.) and their combinations. CLAMOUR is highly language-dependent, whereas the statistical approach is language-independent.

== Applications ==

Compound -term processing isallows importantinformation-retrieval becauseapplications, itsuch allowsas [[search (and other Information Retrieval) applicationsengines]], to perform their matching on the basis of multi-word concepts, rather than on single words in isolation which can be highly ambiguous.

MostEarly search engines simply looklooked for documents that containcontaining the words thatentered by the user enters into the search box (aka. "These are known as [[keyword search"]] engines). [[Boolean search]] engines add a degree of sophistication by allowing the user to specify additional requirements. butFor most users struggle to comprehend and use the necessary syntax (e.g.example, "Tiger NEAR Woods AND (golf OR golfing) NOT Volkswagen)." [[Phraseuses search]]the isoperators easier"NEAR", "AND", "OR" and "NOT" to understandspecify butthat canthese leadwords tomust manyfollow usefulcertain documentsrequirements. beingA missed[[phrase ifsearch]] theyis dosimpler notto containuse, but requires that the exact phrase specified appear in the results.

==See also==

Line 23 ⟶ 32:

== References ==

{{Reflist|30em}}

{{-}}

== External links ==

{{Natural Language Processing}}

*[http://www.conceptsearching.com/ Concept Searching Limited]

*[http://www.webmasterwoman.com/search-engines/phrase-based-indexing.html Webmaster Woman]

{{DEFAULTSORT:Compound Term Processing}}

[[Category:Concept Searching Limited]]

[[Category:EnterpriseInformation searchretrieval techniques]]

[[Category:Information retrieval]]