How it works
AI assistants with search (ChatGPT, Claude, Perplexity, Google's AI answers) read web pages and then recommend tools and cite sources. This is sometimes called GEO, generative engine optimization.
First, they must be allowed to read your site. Each company has a separate search crawler. OpenAI says sites that block OAI-SearchBot won't be shown in ChatGPT search answers. Anthropic's search crawler is Claude-SearchBot, and Perplexity's is PerplexityBot. If you use Cloudflare, also check its AI-crawler blocking settings, which can block them even when robots.txt allows them.
Second, give them something worth citing. A 2024 Princeton study (the GEO paper) found that adding statistics, quotations and cited sources made pages show up much more in AI answers, while keyword stuffing didn't help.
Third, be mentioned elsewhere. AI answers draw on many sites: reviews, comparison lists, Reddit threads, directories. Mentions on those pages help AI recommend you.
Good fit if
- Products in a category people compare ("best tool for X")
- Sites that already have helpful pages
Skip it if
- Your site has no real content yet. Start with "Write pages people search for"
Before you start, get these ready
Tick them off. This page remembers your ticks on this device.
0 of 5 ready
Step by step
- 1
Ask the AIs yourself
Ask ChatGPT, Claude and Perplexity your buyers' questions. Write down which tools they recommend and which pages they cite. That's your baseline.
- 2
Open the door
Make sure the search crawlers can read your site (robots.txt and firewall). Make sure your important pages work without JavaScript tricks and have clear titles.
- 3
Make pages easy to quote
Put plain facts in clear sentences: what it does, for whom, price, limits. Add real numbers, short quotes from customers, and links to sources.
- 4
Get listed where AI looks
Look at the pages the AIs cited. Many are directories, comparison articles and community threads. Get your product listed or mentioned there honestly.
- 5
Re-check every month
Ask the same questions again. Note whether you appear and which pages are cited.
Mistakes that waste your effort or get you banned
- Don't stuff keywords or hidden text. It didn't help in the GEO study and can hurt in Google
- Don't post fake reviews or fake Reddit recommendations. It breaks platform rules and can backfire badly
- Don't block the AI search crawlers if you want to be recommended
Tips
- Blocking training crawlers (like GPTBot) and allowing search crawlers (like OAI-SearchBot) is possible. They are separate
- A clear comparison page often becomes the page AI cites
- Web Scan checks whether AI crawlers can read your site
How to know it's working
- How often you're recommended for your 10 questions (check monthly)
- Visits from chatgpt.com, perplexity.ai and claude.ai in your analytics
- Sign-ups where people say an AI suggested you
Template: robots.txt that lets AI search in
Fill in the [brackets]. Keep it in your own voice.
User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: * Allow: / Sitemap: https://[your-domain]/sitemap.xml
Where this comes from
Checked 4 October 2026. Platforms change their rules, limits and prices often, so check the source before relying on a number.