Getting discovered by search engines
Getting discovered by search engines
New sites no longer need to be submitted to search engines to be listed. A simple link from an established site will get the search engines to visit the new site and spider its contents. It is rarely more than a few days from the acquisition of the link to all the main search engine spiders visiting and indexing the new site.
Naturally, this means that it is good practice to have some means (such as a site map, or plain hypertext links) so that once a spider finds part of a site, it can navigate to the rest. Otherwise, individual, isolated, dead-end pages must be found one-by-one from outside the site; any pages that are not linked to from outside can only be found by links internal to the site.
In addition, creating a robots.txt file will help spiders or robots index only the section(s) of the site you want the public to see. For example, you don’t want your “cgi-bin” folder to be index by the spiders or your administration section (shopping cart). Most of the content in the cgi-bin is dynamic plus spiders do not read perl scripts. Googlebot looks for this file -robots.txt- so it will definitely help your rankings if you have one on your root directory.
For those search engines, like Yahoo, who have their own paid submission, it may save some time to pay a nominal fee for submission.
