Glossary Terms

  • robots.txt for AI Crawlers

    robots.txt for AI crawlers is the practice of using the robots.txt file at a domain’s root to allow or block individual AI crawlers by user-agent token — typically separating training crawlers from the search and retrieval crawlers that determine whether a site can be cited. robots.txt for AI Crawlers in Practice The standard setup treats…

  • llms.txt

    llms.txt is a proposed plain-text file, written in Markdown and placed at a site’s root, that gives language models a curated index of a site’s most useful pages. Jeremy Howard of Answer.AI proposed it on September 3, 2024; it is a community proposal, not an official web standard. llms.txt in Practice The evidence on whether…

  • OAI-SearchBot

    OAI-SearchBot is OpenAI’s crawler for search: it indexes pages so they can be surfaced and cited in ChatGPT’s search features. It is controlled by its own robots.txt token, independent of GPTBot (training) and ChatGPT-User (user-triggered fetches). OAI-SearchBot in Practice OpenAI’s documentation states that OAI-SearchBot is for search and that GPTBot is the separate control for…

  • Google-Extended

    Google-Extended is a robots.txt control token — not a separate crawler — that tells Google whether content its crawlers fetch may be used to train future Gemini models and to ground answers in Gemini Apps and Vertex AI. Google states it does not affect inclusion or ranking in Google Search. Google-Extended in Practice A frequent…

  • PerplexityBot

    PerplexityBot is Perplexity’s crawler, designed to surface and link websites in Perplexity’s search results. According to Perplexity’s documentation, it is not used to crawl content for AI foundation models. A separate fetcher, Perplexity-User, handles requests triggered by a user’s question. PerplexityBot in Practice Perplexity’s documentation says Perplexity-User generally ignores robots.txt rules because a person initiated…

  • ClaudeBot

    ClaudeBot is Anthropic’s web crawler for collecting public web content used in model development. Anthropic runs two other bots alongside it: Claude-User, which fetches pages when a user asks Claude about a link, and Claude-SearchBot, which indexes content to improve search result quality. ClaudeBot in Practice In July 2024, ClaudeBot visited the repair site iFixit…

  • GPTBot

    GPTBot is OpenAI’s web crawler for gathering publicly available content that may be used to train its generative AI foundation models. It is controlled through its own robots.txt token and is separate from OpenAI’s search and user-triggered crawlers. GPTBot in Practice OpenAI documents three distinct user agents: GPTBot for training, OAI-SearchBot for surfacing sites in…

  • Hallucination (AI)

    A hallucination is when an AI model produces information that is false, fabricated, or unsupported by any source — while presenting it fluently and with apparent confidence. AI Hallucination in Practice In *Moffatt v. Air Canada* (2024), a customer asked the airline’s website chatbot about bereavement fares. It told him he could buy a full-price…

  • Knowledge Cutoff

    A knowledge cutoff is the point in time after which a language model has no training data. Anything that happened later is absent from its parametric memory unless the system retrieves it through search or another external source. Knowledge Cutoff in Practice The reported cutoff isn’t the whole story. A Johns Hopkins study, “Dated Data,”…

  • Reranking

    Reranking is a second-stage step in a retrieval pipeline where a model re-scores an initial set of candidate results against the query, keeping only the best few. The first stage is fast but coarse; reranking is slower and more precise, so it runs only on the shortlist. Reranking in Practice In the same Anthropic tests…