robots.txt

[rareyush]

hey frnds what is this robots.txt
and what is its use

rareyush Reviewed by rareyush on 12th Mar 2010. robots.txt hey frnds what is this robots.txt and what is its use http://i42.tinypic.com/2yy41ap.jpg Rating: 5

[Ivan]

it tells bots like google or yahoo search engines which pages they can and can't crawl.
Basicaly - if you have some part of website you don't want google to index, then you can add it to this file and google bot won't visit it.

[EsotorisK]

like u wont like to index ur admincp or memberlist or modcp...

Then u have to add spmething like this :

/admincp
/modcp

[rareyush]

it tells bots like google or yahoo search engines which pages they can and can't crawl.
Basicaly - if you have some part of website you don't want google to index, then you can add it to this file and google bot won't visit it.

like u wont like to index ur admincp or memberlist or modcp...

Then u have to add spmething like this :

/admincp
/modcp

thanx guys

[EsotorisK]

Welcome brother...