Testing Site : Prevent Bots ( google etc...) from accessing

[viruz99]

Hey guys , I am planning to create a site for doing some testing but i don't want search engines etc.. from crawling the site . Any idea how to do that >?

Engine : DLE 8.5 ,

* It should work like just any normal site except being indexed

Thanks

viruz99 Reviewed by viruz99 on 7th Oct 2011. Testing Site : Prevent Bots ( google etc...) from accessing Hey guys , I am planning to create a site for doing some testing but i don't want search engines etc.. from crawling the site . Any idea how to do that >? Engine : DLE 8.5 , * It should work like just any normal site except being indexed Thanks Rating: 5

[XSLTel]

in robots.txt
add

User-agent: *
Disallow: /

[Gavo]

Use robots.txt
http://www.free-seo-news.com/all-about-robots-txt.htm

Or use htaccess to password protect the page.
http://tools.dynamicdrive.com/password/

Or use htaccess so only your IP can view the page.


Put the test site in a folder /testforum , if you do that dont use google toolbar or bots will find it

[viruz99]

in robots.txt
add

User-agent: *
Disallow: /

* I just create a text file - rename it to robots then add those 2 lines in it ?
Then place in root folder for the site ?

[XSLTel]

yes file should be like

http://yourdomain.com/robots.txt

and it should be working just fine

[cyber.crime]

why you not bot crawel ur site viruz ?