llms.txt, Schema and AI Crawlers: A Creator's Setup Guide

You finally did it. You bought your own domain, built a little site with your work on it, and felt like a real professional for about a day. Then someone in a creator group chat asked, "wait, did you block the AI bots or allow them?" and you realised you had no idea. You opened yourname.com/robots.txt and stared at a file you did not write and do not understand. Is ChatGPT allowed in? Is Google? Did your website builder decide for you?

That small file, plus a couple of others, quietly decides whether AI assistants can read your site at all. And the logic here is simple: if you don't get famous, how can AI discover you? Fame gets you talked about, but if the bots that power ChatGPT search, Perplexity, Claude and Google's AI features are locked out of your own site, they can only learn about you secondhand.

This guide is the technical setup, explained for people who are not developers. You will learn what each AI crawler does according to the companies' own documentation, how to write robots.txt rules for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended, how to add schema.org Person and Organization markup with sameAs links, the honest status of llms.txt, and whether a link-in-bio page is enough or you need your own domain.

The setup in one paragraph

Three things decide how machines read your site. robots.txt is a plain text file at the root of your domain that tells crawlers which parts of your site they may visit. Structured data is code in your pages, written in the schema.org vocabulary, that labels what the page is about: this person, these profiles, this business. llms.txt is a proposed markdown file that gives AI agents a curated map of your site. The first one matters a lot, the second helps, and the third is optional. Everything else in this guide is detail on those three, plus a decision about where your home base on the web should live.

None of this replaces having something worth reading. Our guide to generative engine optimization for creators covers the content and reputation side. This page makes sure the door is open when the bots arrive.

Search bots, training bots and user fetchers

The single most useful idea in this whole topic: AI companies run different bots for different jobs. Once you see that, the robots.txt decisions become obvious.

• Search bots crawl and index pages so the assistant can cite and link them in answers. Block these and you disappear from that assistant's search results.
• Training bots collect content that may be used to train future AI models. Blocking these is about how your content is used, and the companies document it as separate from search.
• User fetchers visit a page because a person asked the assistant to, for example by pasting your link. Because a human triggered the visit, several companies say robots.txt may not apply to them.

What each company's documentation says

A few more details from those docs. OpenAI says it can take about 24 hours for its systems to adjust after you change robots.txt, and that each of its settings is independent, so you can allow OAI-SearchBot while blocking GPTBot. Anthropic says its bots honour robots.txt, supports the non-standard Crawl-delay rule, and asks you to add rules for every subdomain you want covered. It also warns that blocking its IP addresses instead of using robots.txt may not work reliably, because it stops the bot from reading your robots.txt at all.

Google is the special case

Google does not run a separate AI search crawler. AI Overviews and AI Mode use the normal Googlebot and the normal Search index, so if you block Googlebot you lose regular Search and AI features together. Google-Extended is a token you can name in robots.txt, but per Google's crawler documentation it has no separate user agent string: Google crawls with its usual bots and checks the token to decide how the content may be used.

There is a trade-off hiding in that definition. Google-Extended covers training and also grounding in Gemini apps, meaning Gemini using Search content to answer a question. If being mentioned inside the Gemini app matters to you, blocking Google-Extended works against that goal. It does not affect AI Overviews or AI Mode in Google Search.

Separately, Google added a setting in Search Console that lets you include or exclude your site from generative AI features in Search. Google announced it in June 2026 and says it rolled out to all sites worldwide on August 31, 2026. Google states that sites that opt out will not receive traffic or impressions from those features. If you want to be discovered, leave it on include.

Robots.txt recipes for creators

Your robots.txt file lives at the root of your domain: yourname.com/robots.txt. Each group starts with a User-agent line naming the bot, followed by Allow or Disallow lines. "Disallow: /" blocks the whole site for that bot. An empty file, or no file at all, means every crawler may visit everything.

Before you edit anything, check what you already have by opening that URL in a browser. Many website builders generate the file for you, and some offer a setting to block AI bots instead of letting you edit the file directly.

Recipe 1: maximum discovery

If your goal is to be found and recommended everywhere, the simplest setup is to allow everything. Your file can be as short as:

• User-agent: *
• Allow: /
• Sitemap: https://yourname.com/sitemap.xml

The asterisk means "any crawler." This lets search bots and training bots in. Most solo creators chasing visibility can stop here.

Recipe 2: discoverable, but no training

If you are happy to be cited in AI search but do not want your work used to train future models, block the training tokens by name and leave everything else open:

• User-agent: GPTBot
• Disallow: /
• User-agent: ClaudeBot
• Disallow: /
• User-agent: Google-Extended
• Disallow: /
• User-agent: *
• Allow: /

In the actual file, leave a blank line between groups. OAI-SearchBot, PerplexityBot, Claude-SearchBot and Googlebot all fall under the final catch-all group, so they stay allowed. Remember the Gemini trade-off above before including Google-Extended. Training data also shapes what assistants "know" about you without searching, so blocking it has a visibility cost of its own, just a harder one to measure.

Recipe 3: what not to do

Blocking every AI-related bot "to be safe" removes you from the exact tools your audience is starting to ask. Copying a long robots.txt from a big publisher is the most common way creators lock themselves out by accident, because publishers often have licensing reasons to block bots that you do not share.

Also remember that robots.txt is a public set of instructions that well-behaved crawlers choose to follow. Anything truly private belongs behind a login, since a badly behaved bot can simply ignore the file.

Check you are not blocking bots by accident

Your robots.txt can be perfect while something else blocks the bots. Common culprits:

• CDN and security settings. Cloudflare, which sits in front of a huge number of sites, sorts AI crawlers into Search, Agent and Training categories. Its July 2026 announcement says that for new domains from September 15, 2026, Training and Agent crawlers are blocked by default on pages that display ads, while Search crawlers stay allowed. Cloudflare also warns that multi-purpose crawlers such as Googlebot, Bingbot and Applebot will be blocked for customers who choose to block Training. Read those settings carefully before you click.
• Website builder toggles. Some builders and plugins include a "block AI crawlers" switch. Check what it blocks: search bots, training bots or both.
• IP firewalls. OpenAI asks sites to allow requests from its published IP ranges for OAI-SearchBot. An aggressive firewall can block it even when robots.txt allows it.
• Login walls and noindex tags. A page behind a login, or tagged noindex, cannot be cited. Google requires pages to be indexed and eligible for a snippet to appear in its AI features.

How to test

1. Open yourname.com/robots.txt and read it line by line.
2. In Google Search Console, use the robots.txt report and the URL Inspection tool on your home page and about page.
3. In Bing Webmaster Tools, verify your site and check for crawl errors.
4. If you use a CDN, check its bot or AI crawler settings and its security event log for blocked requests from the bots you meant to allow.
5. Ask Perplexity or ChatGPT with search to summarise your about page by URL. If it cannot reach the page, something is in the way, although user fetchers follow different rules from search bots, so treat this as a rough test.

Schema.org markup for people and brands

Structured data is a block of code, usually in a format called JSON-LD, that you add to a page so machines can read facts about it without guessing. The vocabulary comes from schema.org, a shared project used by the major search engines. For a creator, two types matter: Person if you are building a personal name, Organization if you are building a brand.

The properties that matter

Why sameAs is the star

Schema.org defines sameAs as the "URL of a reference Web page that unambiguously indicates the item's identity," with examples like a Wikipedia page, Wikidata entry or official website. Google's Organization documentation describes it as a URL of a page on another site with more information about your organization, such as a social media profile, and says you can list several.

For a creator with the same name as thousands of other people, sameAs is how you say "the Jess Park on this site is the same Jess Park on these five profiles." Pair it with links on the page itself, and with each profile linking back to your site, and you have built a consistent loop that both humans and machines can follow. Our guide to personal SEO and owning your name covers the rest of that consistency work.

Where to put it and how to test it

Google recommends putting Organization markup on your home page or a single page that describes your organization, such as an about page, and says you do not need it on every page. A Person block on your home or about page follows the same logic. If you have an about page focused on you, Google's ProfilePage type also fits; its documentation lists "An 'About Me' page on a blog site" as a valid use, with a Person or Organization as the main entity.

Many website builders and SEO plugins can generate this markup from a settings screen, which is easier and safer than hand-writing code. After publishing, test the page with Google's Rich Results Test and the Schema Markup Validator at validator.schema.org.

Keep expectations realistic

Google's guide to its generative AI features says structured data "isn't required" for them and there is "no special schema.org markup" to add, while recommending you keep using it because it helps with rich results. Google's structured data docs also say it does not guarantee that features using the markup will appear. Treat schema as a clarity tool that helps systems connect your profiles. It cannot make anyone cite you.

llms.txt: what it is and its honest status

llms.txt is a proposal, first published in 2024 by Jeremy Howard at llmstxt.org, to add a markdown file at /llms.txt that gives AI agents a short, curated guide to a website: a title, a one-paragraph summary and lists of links to the most useful pages. The idea is that web pages are cluttered with navigation and scripts, and an agent working with limited context benefits from a clean map.

Who uses it

The proposal's own site, now on version 2, says thousands of sites publish the file, that documentation platforms generate one automatically, and that OpenAI, Anthropic and Google's Gemini team publish llms.txt files for their developer docs. It also says Chrome's Lighthouse checks for one as part of its agentic browsing audits. Its heaviest use is software documentation, where coding agents follow the links to find API references.

What the search side says

Google is clear. Its guide to optimizing for generative AI features lists llms.txt among things you can ignore and says Google Search does not use such files, including for its AI features. That matches what Google's Gary Illyes said at a Search Central event in July 2025, as reported by Search Engine Land and Search Engine Roundtable: Google does not support llms.txt and is not planning to.

The other major assistants have not documented using llms.txt to decide what appears in their consumer search answers. Publishing llms.txt files for their own docs is a different thing from reading yours to rank creators. As of this writing, it remains a proposal without a confirmed role in AI search ranking.

Should you make one?

If you run documentation, a software product or anything agents might use, it is a cheap, sensible addition. For a typical creator site, it is optional: it will not hurt, it takes about fifteen minutes, and it should come after robots.txt, your about page and your schema. If you do write one, a creator version might include:

• An H1 with your name or brand.
• A one-line blockquote summary, ideally your consistent bio sentence.
• A "Work" section linking your portfolio, case studies or best posts, each with a short note.
• An "About" section linking your about page, contact page and main social profiles.

Keep it accurate. An outdated llms.txt sends agents to dead pages and old prices.

Link-in-bio sites vs your own domain

Most creators start with a link-in-bio page, and that is a reasonable start. They are free, fast and good at their main job: getting a follower from a profile to the right link. As a home base for being found by search engines and AI, they come with limits.

The core problem is content. An AI answer needs sentences to quote, and a page of six buttons has almost none. Your own site can hold an about page that explains who you are in full, a services page with prices and process, and pages that answer the questions people actually ask. Those are the pages that get retrieved and cited.

The best setup for most people uses both. Make your own domain the home base with real content and your structured data, and point every profile to it. If you love your link-in-bio tool, keep it, and make sure its first link goes to your site. Some tools let you use your own domain for the link page, which keeps the address under your control. Our guide on building a personal brand online covers what to put on that home base.

Your 60-minute setup checklist

1. Open your robots.txt and decide between Recipe 1 and Recipe 2. Make sure OAI-SearchBot, PerplexityBot, Claude-SearchBot and Googlebot are not blocked.
2. Check your CDN and builder settings for AI bot blocking, especially any "block training" option that also catches Googlebot.
3. Verify your site in Google Search Console and Bing Webmaster Tools, and confirm the Search generative AI setting is on include.
4. Write or fix your about page so it states your name, what you do and who you do it for in the first two sentences, and links every profile.
5. Add Person or Organization markup with sameAs links on your home or about page, then run the Rich Results Test.
6. Update every profile to use the same name and bio and to link back to your site.
7. Optionally, add llms.txt with your bio and best pages.
8. Test in the assistants. A week later, ask ChatGPT, Perplexity and Claude about you by name and note what they cite.

Once the technical side is clean, the next lever is being talked about. Our guide on how to get recommended by ChatGPT covers earning the mentions and content that put your name into answers.

Frequently Asked Questions

Should I block GPTBot on my website?

Only if you do not want your content used to train OpenAI's models. OpenAI documents GPTBot as its training crawler and OAI-SearchBot as its search crawler, and says the settings are independent. Blocking GPTBot while allowing OAI-SearchBot keeps you eligible for ChatGPT search answers.

What is the difference between ClaudeBot and Claude-SearchBot?

According to Anthropic's help centre, ClaudeBot collects web content that could contribute to model training, while Claude-SearchBot indexes content to improve the quality of Claude's search results. Blocking ClaudeBot signals your future content should be excluded from training. Blocking Claude-SearchBot may reduce your visibility in Claude's search answers.

Does blocking Google-Extended remove me from AI Overviews?

No. Google's crawler documentation says Google-Extended does not affect inclusion or ranking in Google Search, and AI Overviews and AI Mode are part of Search. Google-Extended controls Gemini model training and grounding in Gemini apps and Vertex AI. To leave Search AI features, Google offers a separate Search Console setting.

Does llms.txt help me get cited by ChatGPT or Google?

There is no confirmation that it does. Google says Google Search does not use llms.txt, including for AI Overviews and AI Mode, and the other major assistants have not documented using it to rank sources in their search answers. It remains a proposal from llmstxt.org, most useful for documentation sites that AI agents read.

Is schema markup required to appear in AI answers?

No. Google says structured data is not required for its generative AI features and there is no special markup for them. It is still worth adding Person or Organization markup with sameAs links, because it helps search engines connect your profiles and can make you eligible for rich results.

Is a Linktree or link-in-bio page enough for SEO and AI?

It is a good start for routing followers, but usually too thin to be cited. You typically cannot edit its robots.txt or add full structured data, and a list of buttons gives an AI little to quote. Use your own domain as your home base with an about page and real content, and point your link-in-bio there.

Do AI crawlers always obey robots.txt?

The search and training bots from OpenAI, Anthropic, Perplexity and Google are documented as respecting robots.txt. User-triggered fetchers are different: OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it, because a person asked for the page. Use a login for anything truly private.