Content deleted Content added
m Reverted 1 edit by Chinamicropump (talk) to last revision by MonAll (TW) |
→Limitations: Fix formatting Tags: Mobile edit Mobile web edit Advanced mobile edit |
||
Line 60:
For example, ''[[The New York Times]]'' has a robots.txt file at https://www.nytimes.com/robots.txt which includes:
Thus, archive requests for URLs within those folders, and any other similarly listed folder of the New York Times website will be rejected.
| |||