Content deleted Content added
→Getting indexed: This edit expands on the limits of search engine indexing. |
|||
Line 52:
The leading search engines, such as Google, Bing and Yahoo!, use [[Web crawler|crawlers]] to find pages for their algorithmic search results. Pages that are linked from other search engine indexed pages do not need to be submitted because they are found automatically. Some search engines, notably Yahoo!, operate a paid submission service that guarantee crawling for either a set fee or [[Pay per click|cost per click]].<ref>{{cite web|url=http://searchenginewatch.com/showPage.html?page=2167871|title= Submitting To Search Crawlers: Google, Yahoo, Ask & Microsoft's Live Search |date= 2007-03-12|accessdate=2007-05-15|publisher=[[Search Engine Watch]]}}</ref> Such programs usually guarantee inclusion in the database, but do not guarantee specific ranking within the search results.{{Dead link|date=April 2010}}<ref>{{Dead link|date=April 2010}}{{cite web|title=Search Submit|url=http://searchmarketing.yahoo.com/srchsb/index.php|publisher=searchmarketing.yahoo.com|accessdate=2007-05-09}}</ref> Two major directories, the Yahoo Directory and the [[Open Directory Project]] both require manual submission and human editorial review.<ref>{{cite web|url=http://searchenginewatch.com/showPage.html?page=2167881|title= Submitting To Directories: Yahoo & The Open Directory |date= 2007-03-12|accessdate=2007-05-15|publisher=[[Search Engine Watch]]}}</ref> Google offers [[Google Webmaster Tools]], for which an XML [[Sitemap]] feed can be created and submitted for free to ensure that all pages are found, especially pages that aren't discoverable by automatically following links.<ref>{{cite web|url=http://www.google.com/support/webmasters/bin/answer.py?answer=40318&topic=8514|title=What is a Sitemap file and why should I have one?|publisher=google.com|accessdate=2007-03-19}}</ref>
[[Web search engine|Search engine]] crawlers may look at a number of different factors when [[Web crawler|crawling]] a site. Not every page is indexed by the search engines. Distance of pages from the root directory of a site may also be a factor in whether or not pages get crawled.<ref name="cho">{{cite web|url=http://dbpubs.stanford.edu:8090/pub/1998-51|title=Efficient crawling through URL ordering|author=Cho, J., Garcia-Molina, H.|year=1998|publisher=Proceedings of the seventh conference on World Wide Web, Brisbane, Australia|accessdate=2007-05-09}}</ref> Additionally, search engines sometimes have problems with crawling sites with certain kinds of graphic content, flash files, portable document format files, and dynamic content. <ref> http://www.anvilmediainc.com/wp-content/uploads/ami_seo_whitepaper_1104.pdf</ref>
=== Preventing crawling ===
| |||