404 Not Found - robots.txt ???
 
 
This forum is locked: you cannot post, reply to, or edit topics.   This topic is locked: you cannot edit posts or make replies.   printer-friendly view    phpMiX.org Forum Index -> mxBB Modules -> mxBB Module: mx_errordocs
View previous topic :: View next topic  
Author Message
markus
[Administrator]
[Administrator]


Joined: 28 Jul 2003
Posts: 1124

PostPosted: Wed Sep 24, 2003 12:44 am    Post subject: 404 Not Found - robots.txt ??? Reply with quote

To remove this 404 error from your logs, you may use file robots.txt to show search engines how to crawl your site. This file should be located in your root dir. ie. http://www.your-domain.com/robots.txt

Sample 1: Disallow everything
Code:
User-agent: *
Disallow: /


Sample 2: Probably usefull for MX-System based sites. Confused
Code:
User-agent: *
Disallow: /admin
Disallow: /cache
Disallow: /images
Disallow: /includes
Disallow: /language
Disallow: /templates
Disallow: /common.php
Disallow: /config.php
Disallow: /login.php
Disallow: /phpBB2/admin
Disallow: /phpBB2/db
Disallow: /phpBB2/images
Disallow: /phpBB2/includes
Disallow: /phpBB2/language
Disallow: /phpBB2/templates
Disallow: /phpBB2/common.php
Disallow: /phpBB2/config.php
Disallow: /phpBB2/groupcp.php
Disallow: /phpBB2/login.php
Disallow: /phpBB2/memberlist.php
Disallow: /phpBB2/modcp.php
Disallow: /phpBB2/posting.php
Disallow: /phpBB2/privmsg.php
Disallow: /phpBB2/profile.php
Disallow: /phpBB2/search.php


For more information about search engines and robots.txt:
http://www.searchengineworld.com/robots/robots_tutorial.htm
http://www.robotstxt.org/wc/robots.html
http://www.google.com/webmasters

Cheers

 
 
_________________
http://www.phpmix.org
 
Back to top
View user's profile Send private message
Display posts from previous:   
This forum is locked: you cannot post, reply to, or edit topics.   This topic is locked: you cannot edit posts or make replies.   printer-friendly view    phpMiX.org Forum Index -> mxBB Modules -> mxBB Module: mx_errordocs All times are GMT + 1 Hour
 
Page 1 of 1


 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum