加拿大华人论坛 加拿大百科Robots.txt
在加拿大
For information on blocking any of these robots using the Robots.txt exclusion standard, see http://www.robotstxt.org/wc/exclusion.html AltaVista Ask Jeeves FAST GoogleInktomi LycosMSN Yahoo Company AltaVista User Agent Scooter-3.0.3 (Many variations. Most contain the word Scooter) Robot.txt Identifier User-agent: Scooter Details Company Ask Jeeves User Agent Mozilla/2.0 (compatible; Ask Jeeves) Robot.txt Identifier User-agent: directhit User-agent: teomaagent1 Details Company Fast Search and Transfer ASA User Agent FAST-WebCrawler/3.4/Nirvana) AKA - Mozilla/4.0 (compatible; FastCrawler3, [email protected]) Robot.txt Identifier User-agent: fast Details http://fast.no/support/crawler.asp Powers Alltheweb.com, Lycos and many smaller search engines Company Google User Agent Googlebot/2.1 (+http://www.googlebot.com/bot.html) AKA: Wget/1.5.3 AKA: Googlebot-image (+http://www.googlebot.com/bot.html) Robot.txt Identifier User-agent: googlebot Details http://www.googlebot.com/bot.html Company Inktomi User Agent Slurp (Slurp.so/1.0 ([email protected]; http://www.inktomi.com/slurp.html) Robot.txt Identifier User-agent: slurp Details http://www.inktomi.com/slurp.html Powers AOL, MSN and many others Company Lycos User Agent Lycos_Spider_(modspider) AKA: T-Rex Robot.txt Identifier User-agent: lycos Details Lycos is powered by the search engine at Fast, we don't know why they continue to operate their own spider Company Microsoft / MSN User Agent MSNbot / MSRbot Robot.txt Identifier N/A Details MSNbot is the development Bot for their new search engine MSRbot is supposed to be a Research Bot Page topic: Search engine optimization, What Are Search Engine Robots?Related Content Free Search Engine Positioning Report
评论
User-agent: *Disallow: /User-agent: googlebotDisallow: cheese.htm# robots.txt for http://www.example.com/User-agent: *Disallow: /cyberworld/map/ # This is an infinite virtual URL spaceDisallow: /tmp/ # these will soon disappearDisallow: /foo.html
评论
http://www.oyster-web.co.uk/inf_robots.htm
·加拿大新闻 马斯克三大战略回应特斯拉未来疑问
·加拿大新闻 保时捷W18：18缸怪兽，优点与缺点一览
·加拿大新闻 炸锅！纯中文广告再次现身加拿大！华人无语：当年列治文的伤
·加拿大新闻 2025款马自达EZ
·加拿大新闻 最新民调：邹至蕙支持率崩盘！多伦多人要求换新市长
·中文新闻 凯蒂·辛德：随着爱泼斯坦的爆料不断升级，菲姬被独立电视台
·中文新闻 肖恩·宾妻子隐藏的健康之战：演员的第五任妻子、40 岁的“战
