An AI search engine does not answer from memory. It rewrites your question into one or more search queries, retrieves pages from a search index, picks the passages that fit best and has a language model write an answer from them, with links to the sources. The technique is called retrieval-augmented generation (RAG), or grounding. That is why your visibility in AI answers depends mainly on whether you are in the right index and whether individual passages on your page can be understood and cited on their own.
In brief
- A language model has two sources of knowledge: what it learned in training and what it retrieves from the web at answer time. Citations almost always come from the second.
- The question is rewritten and often split into several searches. Google calls this query fan-out.
- Passages, not whole pages, are selected and cited.
- Each vendor runs separate bots for training, search indexing and user-triggered fetches. You can opt out of training and still appear in answers.
- No vendor publishes its ranking weights. What is documented is the baseline: access, indexing and readable text.
Two sources of knowledge: training data and live retrieval
A language model is trained on a large body of text up to a cutoff date. What it learned is then stored in the model's parameters. That knowledge is broad but has three weaknesses: it goes stale, it cannot be updated without retraining, and the model cannot point to where a given fact came from.
The fix was described in 2020 in the paper Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks by Patrick Lewis and colleagues. The authors combined the model's built-in "parametric" memory with an external, searchable memory, and named provenance and updating world knowledge as open problems for plain language models. Today's AI search engines use the same idea, except that the external memory is an index of the web.
For a site owner the difference is practical:
| Training data | Live retrieval | |
|---|---|---|
| When is your page read? | During collection for training, months or years before the answer | When the index is built, and sometimes at the moment the question is asked |
| Do you get a link? | Normally not | Yes, the source is shown in the answer |
| How fast does an edit show up? | Not until the next model version | Once the page has been recrawled |
| Which bot do you control? | Training bots such as GPTBot and ClaudeBot | Search bots such as OAI-SearchBot, PerplexityBot, Googlebot and bingbot |
How models weigh sources, in training and at retrieval, is covered in our article on how large language models choose their sources. This article sticks to the mechanics.
From question to answer in five steps
1. The system decides whether to search
Not every question triggers a search. Anthropic describes this most clearly: Claude searches when a request depends on information that is current, changing or outside its training data, such as news, prices or facts about a specific organisation, and answers directly when the request draws on stable knowledge, according to the documentation for Claude's web search tool. Google says something similar: AI Overviews appear only when its systems determine they add value beyond classic search results.
2. The question is rewritten and split up
What you type is rarely what gets sent to the search index. Three vendors say so explicitly:
- Google: AI Overviews and AI Mode may use query fan-out, meaning multiple related searches across subtopics and data sources, which lets them show a wider set of supporting links than a classic search. This is stated in Google's documentation on AI features.
- OpenAI: when ChatGPT search works with outside search providers, it typically rewrites your query into one or more targeted queries, according to OpenAI's help article on web search.
- Microsoft: Copilot Chat sends Bing search queries "consisting of a few words" generated from the prompt. Microsoft's own example: a prompt about a possible acquisition of the company Fabrikam becomes the queries
Fabrikam strategyandFabrikam financials. The example is in Microsoft's documentation for Copilot Chat.
Anthropic states that single-fact queries typically use 1 to 3 searches, while comparative or multi-entity research can use 10 or more. The takeaway: you are not competing for one search phrase. A page that clearly answers a narrow sub-question can be pulled into an answer to a much broader one.
3. Candidates are retrieved from an index
The queries run against an index built in advance by a crawler. Google uses its Search index, Copilot uses Bing, and Perplexity has its own index, which according to the company's technical write-up from September 2025 tracked over 200 billion unique URLs at that time. If your page is not in the index it cannot become a candidate, however good it is.
4. Passages are selected
The model cannot read everything. The system has to cut the results down to the parts that actually answer the question. Perplexity is the vendor that describes this step most openly: documents are decomposed into self-contained spans that are retrieved and ranked individually, retrieval combines lexical and semantic search, and ranking runs in stages where fast scorers filter first and heavier reranker models shape the final set. Anthropic describes a different approach: in newer versions of its search tool, Claude writes code that filters the search results before they reach the model, so that only relevant content gets through.
Google, OpenAI and Microsoft do not publish equivalent detail. But the principle that passages are judged individually explains why self-contained text works better than text that leans on its surroundings:
Hard to lift out: "As mentioned above, the same applies here, though the period is shorter."
Easy to lift out: "Under EU rules, the withdrawal period for online purchases is 14 days, counted from the day you receive the goods."
The second passage makes sense without the rest of the page. The craft is covered in our guide to writing content AI wants to cite.
5. The answer is written and sources are attached
The selected passages are placed in the model's context together with the question, and the model writes the answer from them. A citation is tied to the passage that supports a given claim. Claude's API makes this visible: each citation carries the source URL, the title and up to 150 characters of the cited text, and citations are always enabled for web search. Perplexity states that each answer includes numbered citations linking to the original sources. Copilot Chat also shows the exact search queries that were sent to Bing.
A citation does not guarantee that the claim is correct. The model can summarise a source wrongly or draw a conclusion the source does not support. Click through to the original when the fact matters.
How the products differ
The table summarises what the vendors themselves document, as of November 2026.
| Product | Where do the pages come from? | What is documented about queries and citations? |
|---|---|---|
| ChatGPT search | OpenAI's own crawler OAI-SearchBot and, sometimes, outside search providers | The query is rewritten into one or more targeted queries when outside providers are used |
| Google AI Overviews and AI Mode | Google's Search index | Query fan-out. Answers are shown with links to supporting pages |
| Perplexity | Its own index, with pages crawled by PerplexityBot | Hybrid retrieval, ranking at span level, numbered sources in every answer |
| Microsoft Copilot | The Bing search service | Short generated queries. Copilot Chat shows the queries next to the sources |
| Claude with web search | A search tool whose provider is not named in the tool's documentation. Anthropic also runs its own search crawler, Claude-SearchBot | Claude decides when to search and can search several times. Citations with URL, title and cited text |
ChatGPT search
OpenAI documents four bots. OAI-SearchBot is used to surface websites in ChatGPT's search features. GPTBot collects content that may be used to train OpenAI's models. ChatGPT-User makes fetches that a user has initiated; OpenAI writes that robots.txt rules "may not apply" to it and that ChatGPT-User is not used to determine whether content may appear in search. OAI-AdsBot checks pages submitted as ads. A robots.txt change can take about 24 hours to take effect for search. All of this is in OpenAI's overview of its crawlers. The help article does not say which outside search providers are used for any given answer.
Google AI Overviews and AI Mode
Google is unusually clear: to be eligible as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to be shown with a snippet in regular Search. According to Google there are no additional requirements, and you do not need new machine-readable files, AI text files or special schema.org markup. You limit use with the same tools as for snippets: nosnippet, data-nosnippet, max-snippet and noindex.
A common misunderstanding concerns Google-Extended. It is a robots.txt token with no user agent of its own, which controls whether your content may be used to train future Gemini models and for grounding in the Gemini apps and Vertex AI. It does not affect inclusion in Google Search and is not a ranking signal, according to Google's list of crawlers.
Perplexity
Perplexity searches the web for every question and builds its own index. PerplexityBot indexes pages for search results and respects robots.txt. Perplexity-User fetches a page when a user's question calls for it and, according to Perplexity's crawler documentation, generally ignores robots.txt because the fetch is user-initiated. Neither is used to train foundation models. The details are in our walkthrough of how Perplexity works.
Microsoft Copilot
Copilot grounds its answers in the Bing search service, and Bing's crawler is called bingbot. Since February 2026, Bing Webmaster Tools has had an AI Performance report showing how often your pages are cited in Microsoft Copilot, in AI-generated summaries in Bing and in select partner integrations, along with the "grounding queries" used when the content was retrieved. Microsoft describes the figures as a sample, and the report shows citations, not clicks. See Bing's launch post for AI Performance.
To limit use, Bing introduced two meta tags in 2023 for what was then called Bing Chat: NOCACHE means only the URL, title and snippet may be used, and NOARCHIVE excludes the page entirely from answers and from training Microsoft's generative models. The description is in Bing's blog post from September 2023.
Claude with web search
Anthropic has three bots. ClaudeBot collects content that could be used for training. Claude-SearchBot crawls to improve the quality of search results. Claude-User fetches pages when a user asks a question that requires it. Anthropic states that its bots honour robots.txt and support the Crawl-delay extension, and advises against blocking by IP address because the bots then cannot read your robots.txt. See Anthropic's help article on crawlers.
Which bot does what
| Vendor | Search index | User-triggered fetch | Training |
|---|---|---|---|
| OpenAI | OAI-SearchBot | ChatGPT-User | GPTBot |
| Anthropic | Claude-SearchBot | Claude-User | ClaudeBot |
| Perplexity | PerplexityBot | Perplexity-User | None. Perplexity states that neither bot is used to train foundation models |
| Googlebot | Not covered here | Google-Extended (robots.txt token, no user agent of its own) | |
| Microsoft | bingbot | Not covered here | Controlled with the NOCACHE and NOARCHIVE meta tags |
This split means you can decline training and still appear in answers. A robots.txt that does exactly that:
# Search and user-triggered fetches: allow
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Googlebot
User-agent: bingbot
Allow: /
# Training: disallow
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /
Sitemap: https://www.example.com/sitemap.xml
Whether to allow training is your call. The point is that the decision does not have to affect whether you can be found. More variants are in the guide to robots.txt for AI crawlers.
What the vendors do not tell you
- Ranking weights. No vendor publishes how relevance, authority and freshness are weighed against each other when passages are chosen.
- Number of sources. How many pages are retrieved and how many end up cited varies and is not disclosed.
- Outside indexes. OpenAI says outside search providers are used sometimes, not when. The documentation for Claude's search tool names no provider.
- Stability. The same question can produce different sources at different times, so a single spot check tells you little.
Anyone claiming to know exactly why a particular page gets cited is guessing. What you can do is meet the documented baseline and measure over time.
What this means for your site
- Let the search bots in. Check robots.txt, your firewall and your CDN. A blocked search bot means the page is not in that index.
- Be in both Google and Bing. Their indexes sit behind AI Overviews, AI Mode and Copilot. Verify the site in Search Console and Bing Webmaster Tools and submit your sitemap.
- Deliver content as text in the HTML. Google recommends keeping important content in text form. See the technical factors for AI visibility.
- Write self-contained passages. One paragraph, one point, with the subject spelled out instead of "it" and "this".
- Keep structured data in line with the visible text. That is Google's explicit advice. See the guide to JSON-LD and structured data.
- Measure. Google includes traffic from AI Overviews and AI Mode in the Search Console Performance report under the Web search type. Bing has AI Performance. Methods for the other engines are in the article on measuring GEO results.
To see how your own page handles the baseline, run Visicite's free GEO scan. Among other things it checks whether AI bots are allowed in robots.txt and how dependent the page is on JavaScript.
Frequently asked questions
How do AI citations work?
Once an AI search engine has retrieved pages, it selects the passages that best answer the question and places them in the model's context. The model writes the answer from those passages, and each claim is linked to the passage it rests on. The link you see in the answer points to the page the passage came from. The citation therefore comes from live retrieval, not from the model's training data.
How do you optimise a website for AI search engines like ChatGPT and Perplexity?
Start with access: allow OAI-SearchBot and PerplexityBot in robots.txt and make sure your firewall does not stop them. Then make sure the content is present as text in the HTML, that each passage makes sense on its own and that the page answers specific questions with specific facts. No single measure guarantees a citation, but without access and readable text you are not even a candidate. The full picture is in the guide to GEO.
Do I have to allow GPTBot to appear in ChatGPT search?
No. According to OpenAI, OAI-SearchBot is what controls whether your site can be surfaced in ChatGPT's search features. GPTBot concerns collection for training. You can disallow GPTBot and allow OAI-SearchBot.
Does blocking Google-Extended keep me out of AI Overviews?
No. Google-Extended controls training of Gemini models and grounding in the Gemini apps and Vertex AI. AI Overviews and AI Mode are part of Google Search and are controlled through Googlebot and the snippet rules nosnippet, max-snippet and noindex.
Do I need special markup to appear in AI answers?
Google states that no special optimisations or new markup are required for AI Overviews and AI Mode beyond ordinary good SEO. The other vendors document no requirement for special markup. Structured data can still help systems understand a page, but it is not an entry requirement.