Compound-term processing: Difference between revisions

Revision as of 21:32, 20 June 2008

Revision as of 11:10, 23 June 2008

Line 1:

'''Compound term processing''' is the name that is used for a category of techniques in [[Information retrieval]] applications that performs matching on the basis of compound terms. Compound terms are searchbuilt termsby combining thattwo contain(or more) thansimple one wordterms, for example "triple" is a single word term but "triple heart bypass" is a compound term. For the purpose of information retrieval, it is important to decide which terms to use for searching. In other words, the task is to decide (i) if the whole compound term should be considered as an index and, (ii) which of its components are the more relevant for information retrieval.

In August 2003 [[Concept Searching Limited]] introduced the idea of using statistical Compound termTerm processingProcessing via an article published in INFORMATION MANAGEMENT AND TECHNOLOGY (VOL 36 PART 4). A British Library Direct catalogue entry can be found here: <ref>[http://direct.bl.uk/bld/PlaceOrder.do?UIN=138451913&ETOC=RN] British Library Direct catalogue entry</ref>.

The complete original article can also be downloaded from here: <ref>[http://www.conceptsearching.com/Web/UserFiles/File/Concept%20Searching%20Lateral%20Thinking.pdf] Lateral Thinking in Information Retrieval</ref>.

Further discussion of Compound Term Processing can be found here: <ref>[http://www.statistics.gov.uk/methods_quality/clamour/coordination/wp03.asp] National Statistics CLAMOUR project</ref>. CLAMOUR is a European collaborative project which aims to find a better way to classify when collecting and disseminating industrial information & statistics. In contrast to the techniques discussed by Concept Searching Limited, CLAMOUR appears to be primarily a linguistic approach, rather than one based on statistical modelling. The final project report (dated March 2002) can be found here: <ref> [http://www.statistics.gov.uk/methods_quality/clamour/downloads/Clamour_march2002_final_reportAO.pdf]CLAMOUR Final Report </ref>

Compound term processing is important because it allows search (and other Information Retrieval) applications to perform their matching on the basis of multi-word concepts rather than single words in isolation which can be highly ambiguous.

Compound termTerm processingProcessing is important because it allows search (and other Information Retrieval) applications to perform their matching on the basis of multi-word concepts rather than single words in isolation which can be highly ambiguous.

Most search engines simply look for documents that contain the words that the user enters into the search box (aka "keyword search" engines). [[Boolean search]] engines add a degree of sophistication by allowing the user to specify additional requirements but most users struggle to comprehend and use the necessary syntax (e.g. Tiger NEAR Woods AND (golf OR golfing) NOT Volkswagen). [[Phrase search]] is easier to understand but can lead to many useful documents being missed if they do not contain the exact phrase specified.

Line 11 ⟶ 13:

Techniques for probabilistic weighting of single word terms dates back to at least 1976 and the landmark publication by Stephen Robertson and [[Karen Spärck Jones]]: Relevance weighting of search terms originally published in the Journal of the American Society for Information Science. <ref> [http://www.soi.city.ac.uk/~ser/papers/RSJ76.pdf] Relevance weighting of search terms</ref> Professor Stephen Robertson has stated repeatedly that the assumption of word independence is “patently not justified” and exists simply as a “matter of mathematical convenience”.

Compound termTerm processingProcessing is a new approach to an old problem: how to improve the relevance of search results without missing anything important whilst maintaining ease of use. By forming compound (i.e. multi-word) terms and placing these in the search engine's index the search can be performed with a higher degree of accuracy because the ambiguity inherent in single words is no longer a problem. A search for ''survival rates following a triple heart bypass'' will locate documents about this topic even if this precise phrase is not contained in any document. A [[concept search]] using "Compound Term Processing" can extract the key concepts automatically (in this case "survival rates" and "triple heart bypass") and use these to select the most relevant documents.

In 2004 Anna Lynn Patterson filed a number of patents on the subject of "Phrase based indexing and retrieval" and to which Google subsequently acquired the rights. A full discussion of the patents can be found here: [http://www.webmasterwoman.com/search-engines/phrase-based-indexing.html Webmaster Woman]. The patents themselves can be found online, for example: <ref>[http://appft1.uspto.gov/netacgi/nph-Parser?Sect1=PTO1&Sect2=HITOFF&d=PG01&p=1&u=%2Fnetahtml%2FPTO%2Fsrchnum.html&r=1&f=G&l=50&s1=%2220060031195%22.PGNR.&OS=DN/20060031195&RS=DN/20060031195] US Patent: 20060031195</ref>.

Statistical Compound termTerm processingProcessing is more adaptive than the "phrase based indexing and retrieval" detailed by Anna Lynn Patterson in her patent applications. The "phrase based indexing" is targeted at searching the World Wide Web where Google can utilise their extensive statistical knowledge of common searches to identify candidate phrases. Statistical Compound termTerm processingProcessing is more suited to [[Enterprise Search]] applications where such a priori knowledge is not available.

Statistical Compound Term Processing is also more adaptive than the linguistic approach taken by the CLAMOUR project which considers the syntactic properties of the terms (part of speech, gender, number) and their combination. CLAMOUR is highly language dependent, whereas the statistical approach is language independent. CLAMOUR is a European collaborative project which aims to find a better way to classify when collecting and disseminating industrial information & statistics.

==See also==