Content deleted Content added
No edit summary |
No edit summary |
||
| (25 intermediate revisions by 19 users not shown) | |||
Line 4:
{{Original research|date=March 2021}}
}}
'''Search engine scraping'''
Most commonly larger [[search engine optimization]] (SEO) providers depend on regularly scraping keywords from search engines
The process of entering a website and extracting data in an automated fashion is also often called "[[Web crawler|crawling]]". Search
▲The process of entering a website and extracting data in an automated fashion is also often called "[[Web crawler|crawling]]". Search engine’s like Google, Bing, Yahoo or [[Sogou]] get almost all their data from automated crawling bots.
== Difficulties ==
Google is
Although Google does not take legal action against scraping, it uses a range of defensive methods that makes scraping their results a challenging task, even when the scraping tool is realistically [[Spoofing attack|spoofing]] a normal [[web browser]]:
* Google is using a complex system of request rate limitation which can vary for each language, country, User-Agent as well as depending on the keywords or search parameters. The rate limitation can make it unpredictable when accessing a search engine automated, as the behaviour patterns are not known to the outside developer or user.
* Network and [[IP address|IP]] limitations are as well part of the scraping defense systems. Search engines can not easily be tricked by changing to another IP, while using proxies is a very important part in successful scraping. The diversity and abusive history of an IP is important as well.
Line 30 ⟶ 24:
When search engine defense thinks an access might be automated, the search engine can react differently.
The first layer of defense is a captcha page<ref>{{Cite web|url=https://support.google.com/recaptcha/answer/6081888?hl=en|title=My computer is sending automated queries – reCAPTCHA Help|website=support.google.com|language=en|accessdate=2017-04-02}}</ref> where the user is prompted to verify they are a real person and not a bot or tool. Solving the [[CAPTCHA|captcha]] will create a [[HTTP cookie|cookie]] that permits access to the search engine again for a while. After about one day, the captcha page is
The second layer of defense is a similar error page but without captcha, in such a case the user is completely blocked from using the search engine until the temporary block is lifted, or the user changes their IP.
Line 38 ⟶ 32:
All these forms of detection may also happen to a normal user, especially users sharing the same IP address or network class (IPV4 ranges as well as IPv6 ranges).
== Methods of scraping
{{unref-section|date=July 2024}}
To scrape a search engine successfully, the two major factors are time and amount.
The more keywords a user needs to scrape and the smaller the time for the job, the more difficult scraping will be and the more developed a scraping script or tool needs to be.
Scraping scripts need to overcome a few technical challenges:
* IP rotation using Proxies (proxies should be unshared and not listed in blacklists)▼
* Proper time management, time between keyword changes, pagination as well as correctly placed delays Effective longterm scraping rates can vary from only 3–5 requests (keywords or pages) per hour up to 100 and more per hour for each IP address / Proxy in use. The quality of IPs, methods of scraping, keywords requested and language/country requested can greatly affect the possible maximum rate.▼
* Correct handling of URL parameters, cookies as well as HTTP headers to emulate a user with a typical browser<ref name=":0" /> ▼
*HTML [[Document Object Model|DOM]] parsing (extracting URLs, descriptions, ranking position, sitelinks and other relevant data from the HTML code)▼
▲* Utilizing IP rotation
▲* Proper time management, time between keyword changes, pagination as well as correctly placed delays Effective
▲* Correct handling of URL parameters, cookies as well as HTTP headers to emulate a user with a typical browser
▲*HTML [[Document Object Model|DOM]] parsing (extracting URLs, descriptions, ranking position, sitelinks and other relevant data from the HTML code)
* Error handling, automated reaction on captcha or block pages and other unusual responses{{cn|date=July 2024}}
== Programming languages ==
{{unref-section|date=July 2024}}
When developing a scraper for a search engine, almost any programming language can be used. Although, depending on performance requirements, some languages will be favorable.
[[PHP]] is a commonly used language to write scraping scripts for websites or backend services, since it has powerful capabilities built-in (DOM parsers, libcURL); however, its memory usage is typically 10 times the factor of a similar C/[[C++]] code. [[Ruby on Rails]] as well as [[Python (programming language)|Python]] are also frequently used to automated scraping jobs
Additionally, [[Bash scripting language|bash scripting]] can be used together with cURL as a command line tool to scrape a search engine.
==
When scraping websites and services the legal part is often a big concern for companies, for web scraping it greatly depends on the country a scraping user/company is from as well as which data or website is being scraped. With many different court rulings all over the world.
▲When scraping websites and services the legal part is often a big concern for companies, for web scraping it greatly depends on the country a scraping user/company is from as well as which data or website is being scraped. With many different court rulings all over the world.<ref>{{cite web|url=http://blog.icreon.us/advise/web-scraping-legality|title=Is Web Scraping Legal? |publisher=Icreon (blog)}}</ref><ref>{{cite web|url=https://arstechnica.com/tech-policy/2014/04/appeals-court-reverses-hackertroll-weev-conviction-and-sentence/|title=Appeals court reverses hacker/troll "weev" conviction and sentence [Updated]|website=arstechnica.com|date=11 April 2014 }}</ref><ref>{{cite news |title=App store optimization services |url=https://asoservice.com/ |access-date=17 April 2023}}</ref><ref>{{cite web|url=https://www.techdirt.com/articles/20090605/2228205147.shtml|title=Can Scraping Non-Infringing Content Become Copyright Infringement... Because Of How Scrapers Work?|website=www.techdirt.com|date=10 June 2009 }}</ref>
However, when it comes to scraping search engines the situation is different, search engines usually do not list intellectual property as they just repeat or summarize information they scraped from other websites.
The largest public known incident of a search engine being scraped happened in 2011 when Microsoft was caught scraping unknown keywords from Google for their own, rather new Bing service,<ref>{{cite magazine|url=https://www.wired.com/2011/02/bing-copies-google/|title=Google Catches Bing Copying; Microsoft Says 'So What?'|first=Ryan|last=Singel|magazine=Wired}}</ref> but even this incident did not result in a court case.
==See also==
Line 81 ⟶ 66:
==References==
{{Reflist}}
[[Category:Search engine software]]
| |||