Prompts for robots.txt, Sitemaps, Redirect Maps and Schema

The best ChatGPT prompts for technical SEO ask for a checklist or a line of reasoning before the final answer, force the model to state its assumptions out loud, and get routed through a validator before you publish anything. Use structured prompts for robots.txt with an AI crawler policy, sitemap index planning, redirect maps from a CSV, and JSON-LD schema. Never trust the output on its own. ChatGPT writes a confident first draft, then invents properties, misreads server syntax, and forgets edge cases. Check everything against a real tool before it goes live.

I run HostList, and I see the aftermath of bad AI-generated SEO files every week: a client's sitemap index pointing at URLs that 404, a robots.txt that accidentally blocks the whole site, a redirect map that works on Apache but breaks on Netlify. The prompts below are the ones I actually use with clients migrating hundreds of thousands of pages. Each one is built to expose the model's reasoning, because that's where you catch the mistake before it costs you rankings.

What are the best ChatGPT prompts for technical SEO?

The best prompts are narrow, specific, and demand a checklist before the answer. Vague prompts like "write me a robots.txt file" get you a generic template that ignores your CMS (content management system), your crawl budget, and your actual bot traffic. Specific prompts that name your CDN (content delivery network), your platform, and your constraints get you something closer to usable.

Below are templates for the four jobs people search for most: AI crawler-aware robots.txt files, sitemap index plans for large sites, redirect maps generated from a CSV (comma-separated values file) of old and new URLs, and JSON-LD (a JavaScript-based format for structured data) for Product, FAQPage and Organization schema. Every template ends in a checklist. That checklist is the part most people skip, and it's the part that catches the model's mistakes.

How do you get ChatGPT to write a robots.txt with an AI crawler policy?

You get a usable robots.txt (the file that tells search engine and AI crawlers which parts of your site they can access) by naming every bot you want to control individually, not just using a wildcard. Generic prompts miss GPTBot, ClaudeBot and PerplexityBot entirely, because the model defaults to a template written for Googlebot and Bingbot only.

You are a technical SEO assistant. Write a robots.txt file for a {cms} site at {domain} with {number_of_pages} indexed pages.

Requirements:
- Block /admin/, /cart/, /search/, and any URL with a query string parameter used for filtering.
- Allow all major search engine crawlers (Googlebot, Bingbot) full access.
- Add explicit rules for GPTBot, ClaudeBot, PerplexityBot and CCBot, stating whether each is allowed or disallowed, based on this policy: {ai_crawler_policy, e.g. "block AI training crawlers, allow AI answer engines that cite sources"}.
- Include the sitemap directive pointing to {sitemap_url}.

Before giving the final file, list every crawler you are including and your reasoning for allow/disallow on each one. Then output the complete robots.txt in a code block.

What it gets wrong: ChatGPT regularly forgets that Disallow rules are case-sensitive and path-based, not wildcard by default, so it will write a rule intended to block "/Admin" that does nothing because the actual folder is lowercase. It also frequently confuses "block AI crawlers" with blocking Googlebot's own AI features, which are a different user agent entirely.

Verify it with the robots.txt tester before you upload anything, and check the actual bot traffic hitting your server with the HTTP header checker so you know which crawlers are worth writing rules for in the first place.

How do you plan a sitemap index for a large site?

You plan a sitemap index (a master sitemap file that points to multiple smaller sitemap files) by telling ChatGPT your page count and content types, then asking it to split sitemaps by logical grouping, not just by an arbitrary number. Google's own guidance caps each sitemap file at 50,000 URLs, but splitting by content type makes crawl diagnostics far easier later.

You are a technical SEO consultant. I run a site with {total_url_count} URLs across these content types: {content_types, e.g. "products, blog posts, category pages, author pages"}.

Design a sitemap index structure that:
- Splits URLs into separate sitemap files by content type, not just by count.
- Keeps each individual sitemap under 50,000 URLs and under 50MB uncompressed.
- Recommends a sensible update frequency for each content type based on how often it changes.
- Flags any content type that should probably be excluded from sitemaps entirely (e.g. thin or duplicate pages).

Show your reasoning for the split first, as a checklist, then give me the final sitemap index file structure with example filenames.

What it gets wrong: I have seen ChatGPT recommend a single sitemap for a site with 300,000 product pages, ignoring the 50,000 URL limit completely because it wasn't explicitly stated in the prompt. It also tends to suggest daily update frequency for every content type, which is meaningless noise for pages that rarely change.

Verify it with the XML sitemap validator, which checks file size, URL count, and whether the sitemap actually parses as valid XML before search engines choke on it.

How do you turn a CSV of old and new URLs into a redirect map?

You turn a CSV into a working redirect map by pasting the actual rows into the prompt and specifying the exact server you're deploying to, because nginx, Apache and Netlify use completely different redirect syntax. Asking for a generic "redirect list" gets you something that works nowhere.

You are a server configuration expert. Here is a CSV of old and new URLs (paste as plain text, columns: old_url, new_url, redirect_type):

{paste_csv_rows}

Generate the redirect configuration for {server_type, e.g. "nginx"}. Requirements:
- Use 301 for permanent moves and 302 only where marked temporary.
- Preserve query strings where the old URL had them, unless the mapping row says otherwise.
- Flag any old URL that appears twice, or any new URL that points to another old URL (a redirect chain).
- Group rules logically to keep the file readable.

First list any chains, loops or duplicate rows you found as a checklist. Then output the complete configuration block in the correct syntax for {server_type}.

Take the same redirect map and reasoning from before, but now output it three ways: as an nginx server block, as Apache .htaccess rules, and as a Netlify _redirects file. Note any redirect in the list that cannot be expressed identically across all three (e.g. regex capture groups Netlify handles differently from nginx). List those exceptions before the final code blocks.

What it gets wrong: ChatGPT frequently builds redirect chains without noticing, where old URL A points to old URL B, which itself is being redirected elsewhere. It also mixes up Netlify's _redirects file syntax with Apache's mod_rewrite syntax when you ask for both in one go, producing a file that's valid nowhere.

Verify it with the redirect checker, which follows the chain live and tells you the actual status codes returned, not what the AI assumed would happen.

How do you generate JSON-LD for Product, FAQPage and Organization schema?

You generate usable JSON-LD (structured data written as JavaScript object notation, embedded in a page to help search engines understand its content) by giving ChatGPT your actual field data and telling it to stick strictly to schema.org's defined properties, because it will invent plausible-sounding ones otherwise.

You are a structured data specialist. Generate valid JSON-LD Product schema for this item:

Name: {product_name}
Price: {price} {currency}
Availability: {in_stock or out_of_stock}
Brand: {brand_name}
Reviews: {average_rating} from {review_count} reviews
Product URL: {url}

Only use properties that exist in the official schema.org Product specification. Before the final output, list every property you plan to include and confirm each one is real, not invented. Then output the complete JSON-LD block ready to paste into a script tag.

Generate FAQPage JSON-LD for this list of question and answer pairs:

{paste_faq_pairs}

Rules: each answer must match the visible on-page text exactly, do not paraphrase or shorten it. List the questions you are including as a checklist first, flag any that look like duplicates, then output the final JSON-LD.

Generate Organization JSON-LD for {company_name}, based at {address}, with these social profiles: {social_urls}. Include logo at {logo_url} and a customer service contact point if {phone_number} is provided. Before the output, confirm which properties are required by schema.org's Organization type versus optional, then give me the final code block.

What it gets wrong: I once watched ChatGPT add a "warrantyPromise" property to a Product schema for a client selling digital downloads, a property that doesn't meaningfully apply and was formatted incorrectly anyway. It also loves adding a "reviewRating" nested inside FAQPage schema, which is not a valid combination and will fail validation immediately.

Verify it with Google's own Search Central documentation and read our structured data glossary entry to understand which properties actually matter for rich results, before you paste anything live.

How do you write a prompt for an llms.txt file?

You write a usable llms.txt (a proposed file that gives AI systems a plain-language summary of your site's most important content) by feeding ChatGPT your actual sitemap structure and top pages, not asking it to guess your site's content from the domain name alone.

You are documenting {domain} for AI crawlers and answer engines. Here are our top pages by traffic: {paste_top_urls_with_titles}.

Write an llms.txt file that:
- Summarises what the site does in one paragraph.
- Lists the most important pages with a one-line description of each.
- Notes any content that should not be cited or summarised without attribution.

First list your assumptions about the site's purpose based only on the URLs given, then correct them if I flag anything wrong, then output the final llms.txt.

We cover the full reasoning behind this emerging standard, including what it can and can't do for your visibility in AI answers, in our blog's llms.txt explainer. Read it before you spend time on this, because adoption among AI crawlers is still inconsistent.

How do you use ChatGPT to audit a page's technical SEO before publishing?

You use ChatGPT as a pre-publish check by pasting the page's actual HTML head section and asking it to run through a fixed checklist, rather than asking an open-ended "how's my SEO" question that produces generic advice.

You are auditing this page's technical SEO before it goes live. Here is the head section: {paste_head_html}. Page type: {page_type, e.g. "product page"}.

Check for:
- Missing or duplicate title tag and meta description.
- Canonical tag pointing to the wrong URL or missing entirely.
- Missing hreflang tags if this page has {number_of_language_versions} language versions.
- Any structured data already present, and whether it is well-formed JSON-LD.
- Robots meta tag conflicts (e.g. noindex left in from staging).

List every issue found as a checklist with severity (blocker, warning, minor). Then summarise the top 3 fixes to make before publishing.

What it gets wrong: ChatGPT can't actually render the page or check live HTTP headers, so it will miss a noindex header set at the server level even if the meta tags look fine. It only sees what you paste, and it won't tell you that unless you ask directly.

Verify it with the HTTP header checker, which reads the real server response, catching noindex directives and status codes the AI has no way to see.

Prompt index: what to paste, what you get, how to verify it

Here's the full set in one place, so you can find the right template fast without rereading the whole article.

A few patterns show up across all of these templates. ChatGPT is genuinely useful for first drafts and structural thinking, the kind of work that used to take an hour of reading documentation. It's unreliable for anything requiring live access to your actual server, your actual crawl logs, or the current version of a schema specification.

• Always ask for a checklist or list of assumptions before the final output. This is where you catch invented properties, missed edge cases, and wrong syntax.
• Paste real data wherever possible. Prompts built on placeholders alone produce generic output that ignores your site's actual quirks.
• Never deploy directly from ChatGPT's output. Every template above has a specific validator listed for a reason.

The redirect map templates deserve extra caution. A single wrong regex in an nginx rewrite rule can silently break an entire URL pattern, sending thousands of pages to a 404 without any obvious warning sign. I've seen this happen during a migration where the client only noticed traffic had dropped after two weeks, because nobody checked the redirect checker before going live.

• For robots.txt, always test with a live crawler simulation, not just a syntax check.
• For sitemaps, confirm the file is actually reachable at the URL you declared, not just valid XML in isolation.
• For JSON-LD, run it through validation even if it looks correct, since a single typo in a property name silently drops the whole block from rich results.

If you're choosing hosting partly on how well it handles large-scale technical SEO work, crawl budget and server response consistency matter more than most people realise. Browse the HostList directory to compare providers by their actual HRI score, not marketing copy. For background on how crawlers and CDNs interact, Cloudflare's learning centre and Google's web.dev are both worth bookmarking.

Frequently Asked Questions

Can ChatGPT write a complete robots.txt file on its own?

It can write a syntactically valid file, but it won't know your actual bot traffic, your CMS's real folder structure, or your current sitemap URL unless you supply them. Paste specifics into the prompt, then test the result with the robots.txt tester before uploading it.

Does ChatGPT know the current schema.org properties?

Not reliably. It draws on training data that may lag behind schema.org's actual specification, and it regularly invents plausible-sounding properties that don't exist. Treat every JSON-LD output as a draft and check it against the structured data glossary or schema.org directly.

Is it safe to use AI-generated redirect maps on a live site?

Not without checking first. AI-generated redirects commonly contain chains, duplicate rules, or syntax that works on one server but not another. Run every redirect through the redirect checker in a staging environment before pointing production traffic at it.

Should I block GPTBot and ClaudeBot in robots.txt?

That depends on whether you want your content used to train AI models versus cited in AI answers, which are often different bots with different behaviour. There's no universally correct answer, so decide based on your content's value and write explicit per-bot rules rather than one blanket rule.

What is a sitemap index and do I actually need one?

A sitemap index is a file that lists multiple individual sitemap files, needed once your site passes roughly 50,000 URLs in a single sitemap or mixes very different content types. Smaller sites with a few thousand pages can usually use one flat sitemap without an index at all.

Why does my AI-generated JSON-LD fail Google's Rich Results Test?

Usually because of an invented property, a missing required field, or a mismatch between the schema and the visible page content, such as FAQ answers that don't match the on-page text. Validate every block before publishing, since a single error can drop the entire markup from consideration.

Final thoughts

ChatGPT is a genuinely fast way to get from a blank file to a working first draft for robots.txt, sitemaps, redirects and schema. It's not a substitute for validation, and every template here assumes you check the output against a real tool before it touches production. Treat the model as a drafting assistant with no access to your live server, and you'll use it correctly.