← All posts

4 min read

How we made mexant.com readable to Google and AI

Our own home page sent search engines zero words. What was wrong, what prerendering is, the numbers before and after, and six checks to run on your own site.

Itai Rotem, Mexant

Until this month, our own website had a problem we fix for clients: a search engine asking for the home page got an empty box.

Open mexant.com in a browser and you see a full page: the hero, the services with their prices, the work. But that page was built in the browser, by JavaScript, after it arrived. What the server actually sent was a short HTML file with an empty <div id="root"> and a script tag. We counted the words a program sees when it downloads the page and runs no JavaScript:

Page Words before Words after
Home page 0 970
The work 0 427

Google can run JavaScript, but it does that later, in a second pass, when it has the capacity. Bing does less of it. Most AI crawlers, the ones that feed ChatGPT search, Perplexity and the rest, do not run it at all. To them, our home page was blank.

What prerendering is

A site like ours is made of components: a header, a hero, a price card. Normally the browser downloads those components as code and assembles the page itself. That is fast to build and pleasant to use, and it is invisible to anything that only reads HTML.

Prerendering means we assemble the page once, when we build the site, and save the result as an ordinary HTML file. A visitor, a search engine or an AI crawler asking for the page gets the finished words straight away. The browser then picks up that same HTML and makes it interactive, which is called hydration. Nothing looks different to a person. The difference is that the page now exists before any script runs.

We did it with the tools the site already had. React, the library the site is written in, ships a function that renders a page to a string of HTML. Our build now runs three steps: build the site for the browser, build it again for the server, then run a short script that writes every page's HTML to disk. No new dependency, no new hosting, no monthly fee.

The rest of the list

Words on the page were the biggest fix, not the only one. While we were in there:

  • One table of pages. Every page now has one entry in one file: its address, its title, its description. Everything below is generated from that table, so a new page cannot be added without them.
  • A title and description per page, checked. Ours were typed by hand into each page's HTML, where one forgotten page goes unnoticed. Now the build writes them from the table, with the tags that decide how a link looks when someone shares it, and adds whatever a page was missing.
  • A sitemap that lists the site. Ours listed 1 address. It now lists every page, and it is rebuilt from the table on every build, so it cannot go stale.
  • Structured data. A small block of JSON on each page that tells a search engine, in its own format, who we are and where we are. The home page also lists every service with its real price range, the same figures the service cards show, read from the same file.
  • llms.txt. A plain-text summary of the company, every service and every page, for AI tools that look for one. Before, that address returned "not found".
  • Smaller pictures. The screenshots of our work were loading at full size, about 5 MB on the home page. Phones now get 640 and 1024 pixel WebP copies.

Before and after, measured

We measure with Lighthouse, Google's free page audit, on its mobile setting, which simulates a mid-range phone on a slow 4G connection. Scores are out of 100.

Page Performance SEO Main content shown after
Home page before 65 85 17.3 s
Home page after 98 100 2.2 s
The work before 71 92 11.7 s
The work after 99 100 1.9 s

Performance moves between runs, so we ran the home page three times: 98, 90 and 98. The SEO points we were missing came from five links in a mock-up of a bad website on our home page. They had no address, which a crawler reads as broken links. They are plain text now.

Six checks to run on your own site

You do not need to be technical for any of these.

  1. Look at what a crawler gets. In a desktop browser, right-click your home page and choose "View page source". Search the source (Ctrl+F, or Cmd+F on a Mac) for a sentence you can see on the page. If it is not there, a search engine has to run your scripts to read it, and many AI tools never will.
  2. Ask Google what it has. Search site:yourdomain.com. Every page you care about should be listed, with its own title and a description that makes sense.
  3. Open your sitemap. Go to yourdomain.com/sitemap.xml. It should list every page you want found. If it is missing, or lists two pages on a site of twenty, that is worth fixing.
  4. Check each page's title. The text on the browser tab is the blue headline in search results. If every page says the same thing, they compete with each other.
  5. Run Lighthouse on mobile. Open your site in Chrome, open the developer tools (F12), pick the Lighthouse tab, choose Mobile and run it. Below 90 on Performance or SEO, the report tells you why in fairly plain terms.
  6. Ask who your firewall blocks. Hosting and security services such as Cloudflare can block AI crawlers, and some do it by default. That can be the right call. Just make it on purpose: blocking the crawlers that train AI models is a different decision from blocking the ones that let ChatGPT or Perplexity find you and link to you when someone asks.

If the first check fails, the others matter less. A page with no words in its HTML is a page most of the internet cannot read, however good it looks.