JavaScript and AI Crawlers: Can They Read Your Pages?

Every web page reaches a visitor in two stages. First the server sends a file of HTML. Then the browser runs any JavaScript on the page, and that code can add headings, paragraphs, prices and reviews that weren't in the file. A person with a normal browser sees the finished page and never notices the difference.

A crawler sees whatever it's built to see. One that reads the HTML file and stops there gets only what the server sent. On a site where the server sends a thin shell and a script fills in everything else, that shell might hold a logo, a menu and a line asking the visitor to turn on JavaScript. The copy you paid someone to write lives in the part the crawler skipped.

Custom web apps aren't the only sites with this problem. Page builders, tabbed sections, sliders, FAQ accordions and third party review widgets can all load their text with scripts, so a site that mostly works fine can still hide a few important blocks. You won't know which case you're in until you look at the raw HTML.

Google says it does run JavaScript, in a separate step after it crawls a page. Its JavaScript SEO basics guide describes three phases: crawling, rendering and indexing. Pages that return a 200 status code go into a render queue unless a robots meta tag or header tells Google not to index them, and once Google's resources allow, a headless Chromium browser renders the page and runs its JavaScript. Google says a page may wait in that queue for a few seconds and that it can take longer.

The same guide still recommends server-side rendering or pre-rendering, because it makes a site faster for users and crawlers and because not all bots can run JavaScript. Google's page on dynamic rendering, the older trick of handing bots a pre-built copy of each page, calls that approach a workaround and recommends server-side rendering, static rendering or hydration in its place. So even the search engine that does render JavaScript tells site owners to put the content in the HTML.

Most of the major AI crawlers didn't when they were measured, according to the most detailed public data we could find. Vercel published The rise of the AI crawler using request data from its own network. Its headline finding was that none of the major AI crawlers it studied rendered JavaScript, and it named crawlers from OpenAI, Anthropic, Meta, ByteDance and Perplexity. It also reported that the ChatGPT and Claude crawlers did download JavaScript files but didn't execute them.

The post described two exceptions. Google's Gemini uses Googlebot's infrastructure, which gives it full JavaScript rendering, and Apple's crawler renders JavaScript through a browser-based crawler. Vercel's advice to site owners was to render critical content on the server.

Keep the limits of that evidence in mind. It comes from traffic on Vercel's own network at the time the post was written, and any of these crawlers could change how it works without announcing it. The cautious reading matches what Google already says: some bots can't run scripts, so your most important words shouldn't depend on them. Whether your robots.txt lets those bots in at all is a separate question, and AI crawler access controls covers it.

View the page source and search it for a sentence you can see on the page. The source is the HTML the server sent, before any script ran, so it shows you roughly what a crawler that skips JavaScript gets.

1. Open the page in your browser and pick a sentence from the middle of the main text, something specific like a price range or a town name.
2. Press Ctrl+U in Chrome, Edge or Firefox on Windows, or Cmd+Option+U in Chrome on a Mac, to open the page source.
3. Press Ctrl+F, or Cmd+F on a Mac, and search the source for your sentence.
4. If the sentence is there, a crawler that doesn't run JavaScript can read it. If it's missing, that text arrives by script.

Don't right click and choose Inspect for this test. The inspector shows the page after JavaScript has run, so your text will be there either way. If you want to see what Google in particular ends up with, the live test in Search Console's URL Inspection tool lets you view the HTML Google rendered. That answers the question for Google, and it tells you nothing about crawlers that skip the render step.

Site Scan runs this test on up to 10 pages per scan and records the result as a Measured finding. The check is called Words load without JavaScript, and it's one of the 24 technical facts code counts on every page, next to things like title length, a single H1, schema and tap to call links. Here's what it does.

• It fetches each page the way a plain crawler does and doesn't run the page's scripts.
• It strips out script, style, noscript, SVG, template and iframe blocks, along with the navigation and the site header and footer, so the count reflects the page itself.
• It counts the words that are left, and a page with fewer than 50 fails the check.
• A page that fails is marked Fix now in the report, with the finding Words only load with JavaScript and a short fix for your developer: render the page's words in the HTML from the start.

Measured means code counted it, with no model involved. That matters for everything else on the page, too. Jev, the TypeSafe model that answers 44 typed questions about each page, reads the same server HTML text the code counted. When a page fails this check, its AI judgment findings on proof, local detail and answers will likely look thin as well, and for the same reason. Fix the rendering first and scan again before you rewrite any copy.

The scan is free with no account, with 3 scans a day. It reads your home page plus an even spread of other pages, up to 10 in all. It follows your robots.txt, and when a site shows the scanner a bot check, it stops and reports that page as an error. The report builds in your browser, and you can save it as a PDF or a CSV. For a closer look at a page's HTML, such as crawl directives, images and links, the free SEO, AEO and GEO audit is the separate tool built for that.

Ask for the page's real words to be in the HTML the server sends, before any script runs. Developers have a few standard ways to get there, and the terms Google uses are a good shared vocabulary.

• Server-side rendering builds the full HTML on the server for each request, so the words are there when the page arrives.
• Static rendering builds each page's HTML ahead of time, which suits pages that look the same for every visitor, like most service and town pages.
• Hydration sends the finished HTML first and then attaches the interactive parts with JavaScript, so buttons and forms still work.

The web.dev guide to rendering on the web explains these options and their tradeoffs if your developer wants a reference to work from. Ask them to cover more than the paragraphs. Headings, prices, your phone number, your service area, internal links and structured data all belong in that first HTML response. Review widgets and FAQ accordions deserve a second look, since they're common places for text to load late.

When the change is live, repeat the view source test on a couple of pages, then run a new Site Scan to confirm each page clears 50 words in its server HTML. If a page passes and still doesn't appear where you'd expect, the cause lies somewhere else, and why a page may be missing from AI Overviews walks through the other reasons worth ruling out.