Blocked does not mean hidden
A blocked URL can still appear in results if other pages link to it. To keep a page out of search, allow crawling and use a noindex directive instead.
Related service: technical SEO services.
What is Robots.txt?
Definition
A robots.txt file gives crawlers instructions about which parts of a site they may request.
Also known as:robots exclusion file
← Back to the full glossary