Server-Side Rendering for AI Crawlers: The Technical Floor Nobody Checks

The short version: before ChatGPT, Perplexity, Gemini, Claude or Google's AI Overview can name a Toronto plumber, a Mississauga physio clinic or a Scarborough immigration lawyer, a crawler has to fetch the page, get the words back in the HTML, and get them back fast. Three things break that more often than anything else: content that only exists after JavaScript runs, a server that takes too long to answer, and a Common Crawl block that a CDN switched on without telling anyone. None of these is a ranking factor in the usual sense. They are the floor. Across the local-service sites we monitor, a site that fails one of them is invisible on the engines no matter how good its reviews are, and the owner almost never knows, because the site looks fine in a browser.

Why this is the floor and not a nice-to-have

An AI engine answering "emergency plumber near me in North York" runs a few searches, pulls a handful of pages, reads them, and writes. If your page comes back empty, slow, or forbidden, you are out before the writing starts. Not ranked lower. Out.

Our checklist of what AI engines check orders the work as reach, then read, then trust, and these three failures sit at the first two stages. Schema on a page no crawler can read does nothing. A beautifully written service page that paints in the browser after a script runs is, to most crawlers, a blank page with a logo on it.

The frustrating part for a local-service owner is that nothing looks wrong. Customers see the site. Google has indexed it, because Googlebot renders JavaScript and waits. The AI crawlers mostly do neither.

Failure one: your content is not in the HTML

What goes wrong

Most AI crawlers fetch the raw HTML and read what is there. They do not run your JavaScript. Full single-page apps are the obvious case, and they are rare among GTA trades and clinics. What we actually find on the local-service sites we monitor is subtler, and it comes in pieces:

• Service-area lists and neighbourhood sections built in a JavaScript tab component, so "we serve Etobicoke, Vaughan and Richmond Hill" is not in the source.
• FAQ accordions populated by a script or an embed, which means the exact question-and-answer pairs that engines love to lift are not fetchable.
• Review widgets that pull Google reviews in client-side. The reviews show to humans and vanish for crawlers.
• Pricing tables and booking details loaded from a third-party tool inside an iframe or a script tag.

Each one removes the most citable part of the page while the boilerplate stays visible.

How to diagnose it

Two checks, five minutes.

First, the browser test. Turn off JavaScript in your browser settings, reload your main service page, and read what is left. A spinner, a blank section, or a header with nothing under it means you have found the problem.

Second, the fetch test, which is closer to what a crawler sees:

curl -sL -A "Mozilla/5.0 (compatible; ClaudeBot/1.0)" https://www.example.com/plumbing-toronto | grep -c "Etobicoke"

Swap in your domain and a phrase you know should be on the page. A zero means the phrase is not in the served HTML. Repeat with a phrase from your FAQ and one from a review.

How to fix it

The fix depends on the stack, and we would rather say that plainly than pretend it is a toggle.

• WordPress, Squarespace, Webflow, Wix: the page body is almost always server-rendered. The problem is usually one widget. Replace the JavaScript FAQ with plain headings, and paste your best reviews into the page as text with a date and a first name, alongside the live widget.
• Next.js, Nuxt, Astro or similar: switch the affected routes to server-side rendering or static generation. Someone chose client rendering, and the choice can be reversed per route.
• Plain React or Vue single-page app: this is a real project. Prerendering and static export are the cheaper routes. A rebuild is the honest answer if the site is small, and most local-service sites are.

Whatever you change, re-run the curl test. The words have to appear in the raw response.

Failure two: your server answers too slowly

What goes wrong

AI crawlers are impatient by design. Suganthan Mohanadasan's teardown of crawler fetch behaviour describes engines abandoning slow fetches and citing a faster competitor instead, with time-to-first-byte under roughly half a second as the working threshold. In server logs the abandonment shows up as 499 status codes, which is what nginx records when the client gives up before the response is sent.

Half a second is the first byte, not the full page load. Plenty of Toronto service sites feel fine to a human while the server takes over a second just to start responding, because every hit rebuilds the page from the database with no cache in front of it.

How to diagnose it

curl -o /dev/null -s -w "TTFB: %{time_starttransfer}s\n" https://www.example.com/

Run it five times, against your home page and your two most important service pages. Note the spread. One slow result is noise. Consistent readings above half a second are the problem.

Then search your server or CDN logs for 499. A steady trickle of them from user agents like GPTBot, ClaudeBot, PerplexityBot or OAI-SearchBot is the engines telling you, in writing, that they left.

How to fix it

• Add full-page caching. On WordPress that is a caching plugin or a host-level page cache that serves HTML without touching PHP. This alone usually takes a slow site under the line.
• Put a CDN in front so the first byte comes from a nearby edge node rather than a shared server in another country.
• Upgrade hosting last, once the cache is in place, if the readings are still high.

Re-test with the same curl loop and keep the numbers. Time-to-first-byte drifts back up as plugins pile on, so put the check on a monthly calendar.

Failure three: you are blocking Common Crawl by accident

Common Crawl is a non-profit web archive, and its crawler is called CCBot. Its archive is a major source of training data for large language models. Blocking it does not stop ChatGPT's live search from fetching you today, but it quietly removes your business from the material the next generation of models learns from, and a brand the model already knows gets named far more often than one it has to discover.

Suganthan Mohanadasan flagged that more than half a million sites block CCBot, and that most of them did it without deciding to. CDNs, security plugins and "block AI bots" toggles add the rule by default. Cloudflare announced in 2025 that new domains joining its network would be asked whether to allow AI crawlers, with blocking as the default. A GTA roofer who moved to Cloudflare for the free SSL certificate may be refusing every AI crawler without ever having seen the setting.

How to diagnose it

Open yourdomain.com/robots.txt and look for CCBot, GPTBot, ClaudeBot, PerplexityBot and OAI-SearchBot. A Disallow under any of them is a block you can see. The robots file is only half of it, though. Fetch the page the way the bot would:

curl -s -o /dev/null -w "%{http_code}\n" -A "CCBot/2.0 (https://commoncrawl.org/faq/)" https://www.example.com/

A 200 is fine. A 403, a 429, or a 503 challenge page means something between the bot and your server is refusing it even though robots.txt says come in. Repeat with the other user agents, then check the CDN dashboard for bot-management or AI-crawler settings and any security plugin for a bot blocklist.

How to fix it

Decide what you actually want. If you want citations, allow CCBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended in robots.txt, and turn off the CDN rule that blocks them. If you want to be cited but not used for training, allow OAI-SearchBot and ClaudeBot and disallow GPTBot and CCBot, knowing the trade you are making. The configuration almost nobody has is a deliberate one.

What our audits flag first

When we audit a local-service site for AI visibility, these three checks run before we read a word of the copy. Our free AEO checker fetches the page as a crawler would, tests whether the content survives without JavaScript, and separates a crawler that was blocked from one that simply got no response, because the two have different fixes.

What we see across the local-service sites we monitor, described without counts because we do not publish counts we cannot stand behind: the JavaScript failure is usually partial, a widget rather than the whole page, and it takes out the FAQ or the reviews. The speed failure clusters on older WordPress installs with no page cache. The crawler block almost always traces to a CDN or security setting the owner never opened. Fix them in the order above.

Rarely does a site fail all three. Rarely does a site pass all three on the first audit either.

Where this fits in the bigger picture

Passing the floor does not get you cited. It gets you considered. After that the work shifts to the answer-first passage, the review corpus, the entity signals off your site, and the engine-by-engine differences in what each one grounds on. How to check your AEO score walks through the full readiness picture, and if you would rather hand the measurement and the fixes to us, that is what our SEO and AEO service does.

Start with the floor. It is the cheapest fix on the list, and it is the one that silently zeroes out every other fix you pay for.

FAQ

Does my local-service site need server-side rendering to show up in ChatGPT? Your content needs to be in the raw HTML the server sends, which is what server-side rendering guarantees. Most WordPress, Squarespace, Webflow and Wix sites already do this for the main page body. The usual gap is a JavaScript widget, such as an FAQ accordion or a review feed, whose text is not in the source. Test by disabling JavaScript and reloading.

How fast does my server need to respond for AI crawlers? The working threshold, per Suganthan Mohanadasan's analysis of crawler behaviour, is a time-to-first-byte under roughly half a second, meaning the delay before the first byte arrives, not the full page load. A slow first byte shows up in server logs as 499 status codes from crawler user agents. Full-page caching and a CDN are the usual fixes.

Is blocking CCBot the same as blocking ChatGPT? No. CCBot belongs to Common Crawl, whose archive feeds model training. ChatGPT's live search uses OAI-SearchBot, and its training crawler is GPTBot. Blocking CCBot removes you from a major training input, which affects whether future models already know your business. Many sites block it by accident through CDN or security-plugin defaults.

My site passes Google PageSpeed. Does that mean AI crawlers can read it? Not necessarily. PageSpeed measures how fast a page renders in a browser, and Googlebot executes JavaScript. AI crawlers mostly do not, and they care about the time to the first byte rather than the full render. A site can score well on PageSpeed while serving JavaScript-only content from a slow server. Run the three checks in this post separately.

• $3,000 Growth CreditFree assessment · credit applied to onboarding. See if you qualify →

Want the optimization playbook, not just the platform overview? Get our free ChatGPT Ads cheat sheet, context hints, bid-floor tests, industry readiness, and the measurement stack from practitioners running real budgets.

Quick answers