 |
Q Why do search engines observe the Robot Exclusion Standard? |
Some webmasters have specific files or directories that they do not want search spiders to index. These places are included in a robots.txt file for search engines to check and then skip in the crawl. If a search engine does not observe the protocol, and some don't, then it runs the risk of being banned access to a site completely by a webmaster using his or her server software. Search engines observe the protocol so they don't trudge through places uninvited by the webmasters.
|
|
|
First answer by Joshuaschwarz. Last edit by Joshuaschwarz. Contributor trust: 15 [recommend contributor]. Question popularity: 6 [recommend question]
|
Research your answer: |
Can you answer other questions about web searches and search engines?
|
|
