Our Training Programs

All contents are copyright of their authors.

Robots.txt Disallow All

Robots.txt Disallow All

Abhishek Chadha

3yAsked Dec 7, 2022, 10:26 AM919.NET

Hi, I have a content management system in Sitecore and I updated the robots.txt file for the test environment with below -

User-Agent: *
Disallow: /

but somehow it gets indexed on google and bing how I can overcome this by not indexing the test environment on these browsers?

Know the answer? Post it — somebody with the same question will find it here.

It is the same account you read, post and publish with — and you will come straight back to this page.

Abhishek Chadha3y ago
the strange thing is that I updated my CD site with both capital User-Agent and CM site User-agent and both the sites get indexed
Jignesh Kumar3y ago
Hi Abhishek,

All looks good, just you need to change User-Agent: * to User-agent:

https://searchfacts.com/robots-txt-allow-disallow-all/
Pankajkumar Patel3y ago
Hi Abhishek Chadha,

Remember to use all lower case for the filename: "robots.txt", not "Robots.TXT.

Rules are case-sensitive and should be:
- User-agent: *
- Disallow: /
For more details:
- About /robots.txt
  
  https://www.robotstxt.org/robotstxt.html
- Create a robots.txt file
  
  https://developers.google.com/search/docs/crawling-indexing/robots/create-robots-txt
Hope this will help you!