Check this if you'll be adding code (or don't like smileys).
To confirm that you are human write OK, uppercase:: (by the way we have several filters in place and we monitor everything, so don't loose time)
shortcuts (IE and NS6 only): hit alt+s to send, alt+p to preview, or alt+r to reset
Posted by: Jo-Ann
Posted on: May 12th, 2010, 1:46pm
When I add my site to search engines, one in particular shows this message:
This site does NOT provide the robots.txt file or the robots.txt file is not constructed properly. It is now recommended all sites provide this useful search engine file.
What do I do? Is this necessary ? and how do I do this ? Your assistance is appreciated. Thank You !
Posted by: Support Staff
Posted on: May 12th, 2010, 3:06pm
frankly speaking, robots.txt (not .text, must be .txt) is almost useless nowadays.
it is a text file that you can create with any common text (.txt) file editor such as MSWindows Notepad (or similar software; you have Notepad as it is included in all Windows versions) and you may transfer with EWE / 1Site internal "Generic FTP" feature.
this file does tell the search engine robots that they must not go to the folder cgi-bin, and they also must ignore the folder images, tmp, private (this is just an example; you may not have these folders at all).
If in your web site, you like that the search engines can go everywhere, you can safely avoid putting a robot.txt file
Some robots supports also other directives such as (some examples)
Allow: /folder1/myfile.html Allow: /folder1/foldertobeseen/ (e.g. you disabled folder1 but you want to enable a specific file or subfolder)
Crawl-delay: 10 (number of seconds between two successive crawls)
Visit-time: 0600-0845 (only visit between 06:00 and 08:45 UTC (GMT))
Posted by: Jo-Ann
Posted on: May 13th, 2010, 6:58am
Thanks for lesson on this topic. I appreciate it. It worried me when I got that message ??
Posted by: walterjackson
Posted on: Jan 5th, 2011, 11:15pm
hi, is there a way if i don't want my site to be crawled by spiders or bots?