A file that tells search engine crawlers which pages to crawl or ignore on your website.
The robots.txt file is a text file at the root of your site (e.g., example.com/robots.txt) that tells search engine bots which pages they can and cannot crawl.
Disallow: /admin/ (Keeps bots out of sensitive areas).Blocking a page in robots.txt doesn't stop it from being indexed if someone else links to it. Use the noindex tag for that. Also, one typo (Disallow: /) can de-index your entire site instantly.
Browse related definitions in the same glossary category.
Algorithm
A complex system used by search engines to retrieve data and deliver results for a query. Google uses multiple algorithms to rank websites.
Algorithm Update
Changes made to a search engine's ranking algorithms that can impact website visibility and traffic.
AMP (Accelerated Mobile Pages)
A framework that enables ultra-fast loading of mobile pages by simplifying HTML and limiting scripts.
API Quotas
Hard limits on total API usage over a billing period, often tied to pricing tiers.
API Rate Limits
Restrictions on how many API requests can be made within a time period, requiring throttling and retry logic.
Broken Link
A hyperlink that no longer works because the destination page has been deleted or moved. Damages UX and SEO value.
Understanding "Robots.txt" is just the first step. Our team at TwoSquares specializes in technical SEO and digital strategy, helping brands turn complex concepts into measurable growth.