How search engines work
A search engine crawls, indexes and ranks, in that order and over and over again.
Search engines work in three steps: they crawl the web with bots, they store what they find in an index and they rank that index again with every search.
A crawler such as Googlebot follows links from page to page and reads what is on each page. Whatever is usable ends up in the index, an enormous database that gets updated continuously. Only when a search comes in does the system go through that index and show the pages that give the best answer. That sequence of crawling, indexing and ranking repeats itself constantly, including for pages that have existed for years.
What happens before you see it
- Crawling. Bots follow links and robots.txt rules to work out which pages they may visit. A page without an incoming link, not even from your own site, therefore goes unfound for a long time.
- Rendering. With pages built up through JavaScript, the crawler has to run the page first before it sees the content. More on that in technical SEO for AI.
- Indexing. Only pages judged valuable and unique enough end up in the index. A page that closely resembles another page on your site runs the risk of staying out.
What happens during a search
- Matching. The system searches the index for pages that fit the query, based on the words and the likely intent behind them.
- Ranking. The pages found get sorted on relevance and trust, as explained in how Google decides the order.
- Showing. The results page shows a selection, sometimes topped by an answer box or an AI overview above the ordinary results, which leaves the classic ten blue links less room than they once had.
Want to know whether crawlers can read your site properly? Take the free AI-zichtbaarheidscheck.
Frequently asked questions
How quickly does Google find a new page?
That varies a lot, from hours to weeks, depending on how often Google already crawls your site and whether the page is linked well internally.
What if a page is crawled but not indexed?
That happens when Google judges the page to be too thin, too similar to another page or technically unreachable.
Do AI crawlers such as the one behind ChatGPT work the same way?
Largely, yes, they follow links and read the content too, though they build no searchable index of their own the way Google does. They use the content more often while generating an answer.
Want to know how visible you are in AI?
Take the free AI visibility check, or see how we approach AI visibility for you.