Important
robots.txt controls crawler access; it is not an authentication or security mechanism. Do not use it to protect confidential resources.
Generate crawler directives on the server without shipping the generator logic to the browser.
robots.txt controls crawler access; it is not an authentication or security mechanism. Do not use it to protect confidential resources.